The latest articles on DEV Community by David Ostler (@dro248). https://dev.to/dro248 https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1555703%2Fb1854028-4622-4804-bb41-892883c6a1df.png https://dev.to/dro248 en David Ostler Fri, 31 May 2024 17:31:56 +0000 https://dev.to/dro248/converting-shp-files-to-csv-with-geopandas-7k6 https://dev.to/dro248/converting-shp-files-to-csv-with-geopandas-7k6 <p>Working with geospatial data is really fun! We use this type of data all over the place in data science, and data warehouses have gotten really good at working with it (I'm looking at you, Snowflake!).</p> <p>However, as you get started down the path of using geospatial data, you will likely run into a speed bump: <code>.shp</code> files </p> <h2> What is a <code>.shp</code> file? </h2> <p>A <code>.shp</code> file is a common way of encoding geospatial data in a binary format. This geospatial data will contain geospatial points, polygons (a collection of geospatial points that form an area), and multipolygons (more exotic polygons). </p> <p><a href="https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fomahubfcr80javfeuqwr.png" class="article-body-image-wrapper"><img src="https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fomahubfcr80javfeuqwr.png" alt="Image description"></a></p> <p>A <code>.shp</code> file will be bundled with other files with the same name but different file extensions (e.g., .shx, .dbf, .prj, etc.). In short, you will need all of them.</p> <h2> What is the problem with <code>.shp</code> files? </h2> <p>Unfortunately, most data warehouses don't read <code>.shp</code> files natively. If you are looking to load this dataset into your data warehouse, you are going to need to convert it to a compatible format like CSV.</p> <p>Additionally, a <code>.shp</code> file might use a different coordinate system than the latitude / longitude system you are familiar with (<a href="https://en.wikipedia.org/wiki/World_Geodetic_System#WGS_84" rel="noopener noreferrer">WGS 84</a>).</p> <p>In this tutorial, we will convert a <code>.shp</code> file to CSV and transform a geospatial data to the latitude / longitude coordinate system.</p> <blockquote> <p>The latitude / longitude coordinate system we are familiar with is called WGS 84 EPSG:4326.</p> </blockquote> <h2> Convert a <code>.shp</code> file to CSV </h2> <p>In this example, we'll use <a href="https://geopandas.org/en/stable/" rel="noopener noreferrer">GeoPandas</a> for the conversion.</p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code> import geopandas as gpd # Import .shp file into a GeoPandas DataFrame geopandas_df = gpd.read_file('Grid_100m.shp') # Convert geospatial data to latitude/longitude coordinate system converted_df = geopandas_df.to_crs('EPSG:4326') # Write data to CSV converted_df.to_csv('Grid_100m.csv', index=False) </code></pre> </div> <p>If all goes well, you should now have a well-formatted CSV file that you can inspect with Excel (and load to your data warehouse).</p> <p>Cheers! 🚀</p> datascience python tutorial dataengineering