Many companies still do manual imports, usually a file or files come in and believe me, they can come in from everywhere. At larger enterprise companies, there's a team or individual who is the one that gets sent these files to import. At smaller companies, this is usually a sales operations or CRM administrator who handles this.
These teams or individuals are treated like the most powerful people at a company. I've seen c-suite bow down to these Salesforce admins or sales operations individuals in the hope that the data will be imported quicker. I've seen sales managers tell their reps who are asking for more data to go outbound to, that they will have to wait until Dave (not a real person), the importer, decides their data is worthy of importing.
Why the importer is held in such high regard
Why does this happen? To break it down into two perspectives, and I have been on both sides of it, we will look at the importer role first.
Importing data into a CRM is tricky business, especially with the complexity around CRMs now and all the different conditions that come with them. You have to make sure the data you are working with is all organized and formatted correctly, complex field mapping across tons of custom fields which are now available in a CRM, every record assigned to the right owner, and make sure you are selecting the right options to not overwrite anything important already in the CRM. Then one field mapping is slightly off or the import fails, and you have to start over again from the beginning.
Meet Dave
Dave, and we'll use him as an example, is usually an experienced operator. Dave knows CRMs and has likely been in that role for a number of years. Companies don't let individuals fresh out of university have the keys to their entire platform which runs the business.
Dave gets files sent in from marketing, sales, GTM divisions and anyone else that wants their data importing. This is where the challenge arises for Dave, and the bottleneck. Dave will likely have a backlog of files waiting to be imported and will usually prioritize which files go in based on when they were submitted, to keep things fair. Everyone in the company will be aware they need to get files across and will have an idea of roughly how long they have to wait until the data is in the CRM.
Every CRM has its own particulars
CRMs don't all behave the same, and the differences show up most at import time. Salesforce's Data Import Wizard takes 50,000 records at a time, so a bigger file means splitting it up or moving to Data Loader, a separate desktop application. HubSpot imports every row as a brand new record if your file has no unique column it recognizes, so a list you know is clean ends up doubled. Pipedrive wants custom fields created before the import rather than during it, and only a global admin can reverse an import, within 48 hours. There's loads of issues that can arise, and they are all things Dave has to hold in his head at once. If he's moving between two CRMs, he's juggling both sets, which is covered in Risks of migrating your CRM.
Looking at the file first
Before the import can even happen though, Dave must actually look at the file. Most of the time files that are sent aren't clean, ready to be imported data; they are usually a mess which has come from a marketing event or from a data provider. Most businesses have never settled whose job it is to clean that file before it gets to Dave, which is what Cleaning data: what it means, and who ends up doing it is about.
This is where Dave has a decision: import as much as possible and spend a few hours cleaning up the file to capture more data, or import what's possible in its current form. Either way, Dave gets a bashing from the reps who claim their data is bad, or loses hours on one file. With a backlog of files to go in already, most of the time it's a balance for Dave on capturing as much as possible without spending hours cleaning the file, and you can forget verifying it.
So usually there's a data loss in what's been captured between the file and the CRM. It's just an accepted practice of the data import problem.
The other side of it
Now from the other perspective of those sending files to Dave. Because a lot of these have never even seen an import page on a CRM, they have no idea the magic Dave does to get the data that's on a spreadsheet in front of their faces ready to go in the CRM. This means that they have no idea what would make Dave's life easier and what they could do before sending the file.
I have seen companies implement an education process whereby any files being sent to Dave have to be in a very specific format and have the exact columns in the exact right order for it to be imported. While this does solve the issue marginally, suddenly every person sending a file must constrain what could be a rich data list into columns which may not capture all that they wanted. While the education piece works and makes Dave's life easier, it still ends in the same result, which is a loss of data.
The three routes companies take
- Have a team of people doing the imports, screening and cleaning the files before they are imported. This works at mass volume and when the budget is there. However, it usually means giving extra responsibilities to sales ops, RevOps or CRM admins and adding to their workload.
- Automate the system where data links directly from one place straight to the CRM through an API or similar. This is usually the solution many companies adopt when going through any digital transformation. Suddenly, you can get that data from a provider straight into your CRM and cut Dave out of the equation (sorry Dave).
- Use AI to do the import process for you. This is like the more SaaS based automation but takes it a step above automating the system directly, whereby you use AI workflows to adapt the data coming in and then it gets imported to the CRM. This is also what a lot of the experts are doing now, using AI transformation on data as part of the automation project.
The second one does present the same issues. When it comes to systems mapping data to each other, systems aren't really set up to send and receive data in the right formats. So what you end up with is usually a worse result, where you click one button on one system, let's say a data provider, and then only half of that gets into the CRM and it turns out it's overwritten an enterprise account's contact and taken all of the notes off it.
This is where the experts usually come in to do a data quality audit and make sure the systems are speaking to each other in the right way. It does work, but it takes months, it isn't cheap, and at the end of it you've handed the keys over to those experts, who charge to maintain it and adapt it when you need them to. Should you automate your lead flow? goes through the costs of that in more detail.
The third one can work extremely well, where you have a system that adapts to different file types and levels of information and can even enrich that data before it enters the CRM, a job The hidden gap in migrating your CRM data covers in full. However, it presents a new list of challenges which we haven't covered before.
Three challenges with putting AI in the middle
First, while AI has become increasingly advanced over the years, it still struggles with spreadsheets. It has a habit of inventing data, or saying it has done something when it hasn't. The reason it struggles is because of the volume of data there is and can be on a spreadsheet. AI can work well on a lower volume spreadsheet and do it exactly as you want, however ramp up the volume and this is where it struggles and starts inventing values or only doing half a job.
Second is the cost. Parsing a large spreadsheet uses tokens. The AI has to read and understand each value and what everything means before it can do the transformation, then actually do the transformation, then send it on its way. Tokens aren't cheap at volume, and this is how companies get saddled with a huge usage bill they weren't expecting.
Third, where your data actually goes. Handing your incoming data to an AI to transform and sort it means a third party is processing your customers' personal data, which makes them a sub-processor. So you need to check their contract: what they keep, for how long, where it sits, and whether any of it gets used to improve the service. Ask, and get the answer in writing. None of this is legal advice and your own agreement with them is what settles it.
That goes for any tool, including ours: find out what leaves your workspace, and get them to show you, not just tell you.
So how is the data import problem solvable?
It's by combining the three together. Morphing Dave, the automated system and AI together so that each one has a role to play and doesn't overstep. That's how we've built Operelio:
- Dave is the toolkit. 17 tools for the cleaning and verifying Dave used to do by hand: merging files, fixing headers, removing duplicates, checking email addresses. None of them use AI, so nothing gets made up. The CRM Formatter maps your file to your CRM's fields and shows you exactly what will happen before anything is imported, rather than handing you an error file afterwards.
- Trails is the automation. Files arrive in one place by email, upload or API, get cleaned and verified the same way every time, and go into the CRM on a schedule you set.
- Bridgeant is the AI. It runs the tools rather than reading every value in your file. Out of a trail it can plan a cleanup and run it for you, once you've seen the plan and confirmed it. Inside a trail it sits just before the CRM and holds anything you've told it shouldn't land. It can stop a push, but it can never force one through.
Out of a trail, you run the cleanup and Bridgeant gets the push ready for you to send. In a trail, it all runs on its own, so Bridgeant checks each push before anything reaches the CRM.
When Bridgeant reviews a push, what leaves your workspace is counts, column names and CRM field names, plus the instructions you typed yourself. None of the actual data in your file is sent.
Why all three work well together
The reason all three work well together is we are taking the best parts of each system. Dave's level of experience and knowledge, the automation's consistency and reliability, and the AI reasoning to react, flag and adapt when things aren't going right.
Put them together and each one covers what the other two can't, in something repeatable and easy enough to use that it gets to the end goal. Solving the data import problem into CRMs.
Originally published on the Operelio blog. Is there a Dave where you work, or are you Dave?
Top comments (0)