FileInput Question

I have a datagrid and each record has an uploaded binary file as well as the file name.
When the datagrid loads it was taking a bit of time, not surprised if it was loading the files (10MB each record) in the datagrid.
When I removed the column for the binary files the datagrid does not load any faster.
Is the datagrid still loading these? Is there a way to exclude them from the load event and then just fetch them when the download link (file name) is clicked using an invoke method or something like that?