Course Data Algorithm

I'm currently in the process of completing an advising application for production use at the university I work at. It's a full stack app built using Ruby on Rails and React. Since I don't know if I can get access to Banner, a typical software package you'll see in higher ed that's used to handle all the data related to course loading, registration, etc., I decided to build out the API for my app to include handling course and core curriculum data. Unfortunately, that meant that the data I had to use was not in a format that could easily be seeded to the backend for serving by the API. The following is a brief explanation of how I've begun to prep the data for seeding to the DB.

The data was obtained through a crawl on our university's bulletin website. I will need to build out a more complete web crawler to get all the course data, but for now I simply worked in the browser console to get the data I needed. First thing was to get data in a form that I could use, and that was as simple as the following code:

Array.from(document.querySelector('#DataListWorld').querySelectorAll('a')).map((course) => {
    console.log(course.innerText)
})

Yes, it's a total hack job outputting to an array that I copy and pasted, but I've found the best way to start is to being simply and then work to develop more complexity as necessary.

Next step was to take the copied array and output it in a format that I could then use to store in the DB. I'm using JSONB columns on PostgresSQL (which is brilliant by the way), so the data needs to be stringified to be used in a format that can be saved to the PostgresSQL column.

There are a couple of things that I needed to do to prepare the data for seeding to data column. Each array of courses needed to add keys and an additional value in the following format:

dataListPhysical = ['ASTR 1010', 'GEOG 1111']
// to the following format
{ Physical: [ { name: 'ASTR 1010', credit_hrs: 3 }, { name: 'GEOG 1111', credit_hrs: 3 } ]

The object above would then serve as the core curriculum requirement for Physical Science along with a list of all eligible courses. So, the algorithm needs to take a list of the names of the core areas and set it as the name of the object, then create an array of objects (or a hash in Ruby) which includes the key value pairs course_code => 'course', credit_hrs => 0

Let's walk through the code for the algorithm.

First, I set two arrays, one an array of the course arrays let dataLists = [dataListFoundation, dataListSocial, dataListHumanities, dataListWorld, dataListQuantitative, dataListLife, dataListPhysical] the second a list of the core area names as an array of strings let coreAreas = ['Foundation', 'Social', 'Humanities', 'World', 'Quantitative', 'Life', 'Physical'].

Next is needed to iterate through each name of the core area:

coreAreas.forEach(area, i) => {
 // iterate through the dataLists map 
   let list = dataLists.map((list) => 
   // iterate through each array of courses to set 
      correct key/value pairs
        return list.map((course) => {
          return { course_code: course, credit_hrs: 3 }
        })
   // create new array for each core area that includes 
   the mapped array of courses with their new key/value 
   pairs by mapping the corresponding items in the data 
   list to the corresponding core area name by index 
   (i)
  let coreCourseList = {[area]: list[i]}
   // write each core course list object to file
   fs.writeFileSync(`${area}.json`, coreCourseList, (err) => {
     if (err) throw err;
     console.log('The file has been saved!');
   })
}) 

The above produced the following output:

{"Foundation":[{"course_code":"ENGL 1101","credit_hrs":3},{"course_code":"ENGL 1101E","credit_hrs":3},{"course_code":"ENGL 1101S","credit_hrs":3},{"course_code":"ENGL 1102","credit_hrs":3},]}

Not anything fancy, but it got the job done!