DEV Community

Help gathering data

Paul on February 07, 2019

I don't even really know where to start, so I figure I'll just throw this out there and see where it goes. My daughter is disabled. She's in sport...
Collapse
 
theaccordance profile image
Joe Mainwaring

I've had ideas wither away because they lacked access to the data sources required to drive their purpose. It's frustrating at times, even more so when you know the datasets exist.

PageScraping might work, but I do wonder how complex that might get. It's one thing to scrape a product catalog for a vendor, and another to parse independent websites for possibilities of data.

A couple alternative approaches:

  • Outsource it to an offshore resource to compile
  • Create a means for community contributions, and run a campaign to get an initial dataset populated by reaching out directly to the orgs and asking them to fill out a survey or something to capture the data
Collapse
 
dmfay profile image
Dian Fay

The first step is getting as many programs and resources into the system as possible. Scraping the public link dumps you have is a perfectly reasonable start: some data is better than no data, and once you have records, you can start attaching tags to them. BeautifulSoup is the last dedicated scraping tool I've used but you should be able to get a list of links and descriptions with a little console JavaScript too.

What I'd do after getting the data on what programs simply exist is to set up a form where people can submit reports that such-and-such program serves this-or-that need (and therefore needs this-or-that tag). Some manual validation or cleanup will probably be necessary but it's crucial to get as many reports on as many programs as possible: any individual report can be incomplete or even incorrect, but the more you have the more you reliably know about any given program. Then it's a matter of generating publicity: blog, post on social media, explore partnering with disability support and activist organizations or getting sponsorships.

The Wine AppDb is an example of a website that does something similar to build a community-driven database.

This sounds like something that needs to exist. Good luck!