I think the gravel doesn't make any issue with the drainage. The pipe will be fit inside the gravel since the gravel is dry and firm in the absence of moister. The presence of water may result in the pipe dipping inside the gravel and causing a difference in the drainage level at the crawl space. If possible you can add another layer of fabric and cap the gravel to a certain height to
resolve the drainage issue.