My Journey with the auto_uploader Project — Detailed and Professional Version
This doc is a revised and more detailed version of my daily notes on the auto_uploader project. I kept my personal tone, but made the explanations more precise and technical, added more code examples, and linked to reliable sources (Python docs, StackOverflow). My goal is for the reader to understand not just the technical challenges, but also the practical lessons and my personal growth. Nothing's exaggerated – it's all based on real experience.
Project Goal: What Did I Want to Build?
auto_uploader is a project to link a local folder to Google Drive, monitor changes (file creation, edits, deletions), and automatically upload them. My practical aim was to save time on manual uploads – like for backing up project code or work docs that change all the time.
Skills Used:
- Python programming
- API integration (Google Drive API)
- OAuth 2.0 authentication
- File system monitoring
- Network basics (like checking connections)
- CLI design
- Error handling and logging
- Config management
Tools:
- Python 3.x
-
watchdog(for file monitoring) -
argparse(for CLI) -
pickleandjson(for data storage) -
osandsocket(file ops and networking) -
logging(for logging) -
google-auth-oauthlibandgoogle-api-python-client(Google APIs)
Initial Assumptions:
- Stable internet
- Valid Google account
- Google Drive API enabled (test mode)
- Correct folder paths entered
-
credentials.jsonfile downloaded (from console.developers.google.com) - API permissions in test mode (good for MVP, but should be published for production)
Planning: I aimed to finish in 4 days (Day 1: structure, Day 2: API and OAuth, Day 3: monitoring and CLI, Day 4: testing and debugging). I rated the technical value at 3/5 – not as complex as ML, but more challenging than simple scripts (compared to tools like rclone).
Errors and Lessons: Learning with Code Examples
During development, I hit common Python errors that each taught me a big lesson. Each section has code and links to sources.
1) AttributeError: 'NoneType' object has no attribute 'reason'
This error happened when a function (like an HTTP request) returned None instead of the expected object, and I tried to access reason. Solution: Use SimpleNamespace to simulate it.
from types import SimpleNamespace
mock_response = SimpleNamespace(reason="Connection failed")
print(mock_response.reason) # Output: Connection failed
Lesson: This error often ties to HTTPError (subclass of URLError). URLError is for connection issues (like DNS failure), and HTTPError for status codes (like 404, 500). HTTPError is also file-like with a read method (source).
import urllib.error
try:
import urllib.request
urllib.request.urlopen('http://example.com/bad')
except urllib.error.HTTPError as e:
print(f"HTTP Error: {e.code}, {e.reason}") # e.g.: 404, Not Found
except urllib.error.URLError as e:
print(f"URL Error: {e.reason}") # e.g.: No host
2) builtins: The Heart of Python
For testing, I needed to patch open, and the docs suggested builtins.open. builtins is Python's built-in module that holds default functions and classes (like open, print, len) – in Python 2, it was __builtin__. Everything in Python is an object – even int or str (source).
import builtins
f = builtins.open('test.txt', 'w')
print(type(f)) # <class '_io.TextIOWrapper'>
f.close()
Lesson: Be careful overriding open locally, as it might affect builtins.open.
from unittest.mock import patch
with patch('builtins.open', side_effect=IOError("Fake error")):
try:
open('test.txt')
except IOError as e:
print(e) # Fake error
3) Why mock_open?
open is a context manager with __enter__ and __exit__ methods. mock_open from unittest.mock simulates files without real I/O. Note: Some libs expect bytes (like mode='rb').
from unittest.mock import mock_open, patch
with patch('builtins.open', mock_open(read_data=b'data')):
with open('file.bin', 'rb') as f:
print(f.read()) # b'data'
Lesson: mock_open is key for I/O tests, but check file modes.
4) AttributeError: module 'sys' has no attribute 'args'
This was a typo (args instead of argv). sys.argv turns command-line args into a list. An attribute is anything accessed via . on an object.
import sys
print(sys.argv) # ['script.py', '--path', '/folder'] if run as: python script.py --path /folder
Lesson: Typos waste time – double-check.
5) patch vs patch.object
patch works on import paths (like 'module.func'), and wrong paths give AttributeError. patch.object works directly on imported objects and is safer (source).
from unittest.mock import patch
import mymodule
with patch.object(mymodule, 'func', return_value=42):
print(mymodule.func()) # 42
Lesson: Use patch.object for more precise tests.
6) TypeError: a bytes-like object is required, not 'str'
str is Unicode; bytes is raw binary data (0-255). Some APIs (like binary files) need bytes. Convert with encode/decode (source).
with open('file.bin', 'wb') as f:
f.write(b'data') # Correct
# f.write('data') # Error: TypeError
Lesson: Literals like b"data" create bytes directly; encode converts str.
7) What’s a Literal?
A literal is a value whose type is clear right in the code, like 52 (int), "abc" (str), b"data" (bytes), [1, 2] (list). Non-literals like input() have types determined at runtime.
8) Mock, MagicMock, spec/spec_set
Error 'mock' object is not subscriptable happened because Mock lacks __getitem__. MagicMock has these magic methods but is permissive and might hide API errors. spec_set is stricter.
from unittest.mock import Mock, MagicMock
m = Mock()
# m['key'] # Error: not subscriptable
mm = MagicMock()
mm['key'] = 'value'
print(mm['key']) # value
Lesson: Use spec_set for accurate tests:
m = Mock(spec=dict)
# m.fake_method() # AttributeError
9) Magic Methods and Context Managers
Magic methods like __len__, __getitem__, __enter__, __exit__ define Python's internal behaviors. __exit__(exc_type, exc_val, exc_tb) handles exceptions in context managers.
class MyContext:
def __enter__(self):
return self
def __exit__(self, exc_type, exc_val, exc_tb):
print(f"Exit: {exc_type}")
with MyContext():
raise ValueError("Test")
# Output: Exit: <class 'ValueError'>
10) RecursionError: maximum recursion depth exceeded
This hits when a function calls itself without an exit condition. Python's default limit is about 1000.
def infinite():
infinite()
# infinite() # RecursionError
Lesson: Add exit conditions or use sys.setrecursionlimit (cautiously).
11) Circular Import
When modules import each other in a circle, you get partially initialized module error. Solutions: lazy imports, restructure, or importlib (risky).
def my_func():
import heavy_module
heavy_module.do_something()
Lesson: Design modules with one-way dependencies (source).
12) Professional Debugging
logging beats print. Advanced tools like pdb or IDE debuggers (e.g., PyCharm) are recommended.
import logging
logging.basicConfig(level=logging.DEBUG, filename='app.log')
logging.debug("Value: %s", var)
import pdb
pdb.set_trace() # For interactive debugging
README: The Project's ID Card
A README should specify the audience (devs, users, DevOps), features (like simplicity), and limitations (like setup difficulty). Add badges (e.g., coverage).
# Auto Uploader
Automate file uploads to Google Drive.
## Limitations
- Needs manual `credentials.json`.
- No auto-retry for internet drops.
Reflections: MVP and Final Lessons
The project worked, but had flaws like tough setup (credentials.json needed) and test conflicts (run tests before config). The repetitive testing (mock/patch) wore me out. Big lesson: Do TDD from the start, and "being strict" differs from "having essence" – code should carry your thought signature.
Recommendations:
- Set up
loggingfrom day one. - Write README with a clear audience.
- Use TDD for simultaneous testing.
- Design modules without circular dependencies.
- Use advanced debug tools (like pdb).
Appendix: (Original Text of My Daily Notes on the Project)
Project Notes on auto_uploader
I'm working on a project called auto_uploader. The goal was to link a folder on my computer to Google Drive and monitor and apply changes there. It was mostly to save time on uploading files. The skills I used were python programming, API-integration, OAuth 2.0=Authentication, File-system-monitoring, Network-basic, commandlineinterface(cli), errorHandlinglogging, configuration-management. And tools like python 3.x, watchdog, argparse, pickle, json, os, socket-logging, google-auth-oauthlib, google-api-python-client. With assumptions like: user has active internet, user has a valid Google account, Google Drive API is active and valid, user enters folder paths correctly, user has API permissions in test mode, downloaded credentials.json. And I planned to wrap it up in four days. I rated the technical value at three out of five. It didn't need a ton of energy, but I was interested in getting it going.
So far in the project, I ran into errors that were pretty common, and I decided to write them down. Like the error AttributeError: NoneType object has no attribute 'reason'. What I got from this error was that the func I gave None to didn't have that attribute, meaning none didn't have it. I had to simulate that attribute for it. I decided to use SimpleNamespace from the typing module. Out of curiosity, I searched StackOverflow and saw someone getting this error for trying to get an attribute from HTTPError, and I dug into it a bit. Where does HTTPError come from? This error class inherits from URLError class. I looked for the difference and found out URLError happens because of failed DNS or server, but HTTPError because of 404, 500 errors.
Next, I bumped into the concept of builtins while writing a test, and the doc said to use this for open. I wasn't sure what builtins was yet. From what I saw, in the project I had to use builtins for input too to patch it. So what is builtins? It's a built-in Python module that in older versions was called builtin. This module includes all the built-in functions and classes that don't need import. Meaning when we write with open('test.txt', 'r'): data = f.read(), we're actually calling builtins.open('text.txt', 'r'), and the result is a file object. Okay, what exactly is an object? Then I realized in Python everything is an object like list, file, str, int, ...
Why do we use mock_open for mocking open? I was thinking about it and in searching for the answer, I found that opening a file is a contextmanager, meaning it has output methods that are not simple, they're complex. So unittest takes help from open-mock to mock or simulate the contextmanager and its methods, and mock_open is only for files.
I got to AttributeError: does not have attribute 'args'. Actually, it was a typo in my code, it was argv, but it made me study Attribute a bit. First off, I learned Attribute means anything connected to a Python object and we access it through .. It's a feature of that object. This error is saying the sys module doesn't have the args attribute. You need to use argv, which you can find out by searching.
Now the question comes up, what does sys.argv do? Basically, when a Python script is run from the command line, it turns it into a list that includes the function's arguments. For a moment, I wondered why I didn't do sys.argv with builtins, and saw that sys is an import module. I don't know why sometimes I think less.
In continuing, I was checking and my eye caught patch and patch.object. Question came up, what's the difference between these two? Both are for mocking and simulating, but they differ in logic. The difference is patch simulates on the function path, but patch.object directly on the object. That's why for patch.object you need to import the module you want. Overall, patch.object is considered safer because it works directly on the object and the address and import are correct, but plain patch might hit AttributeError because the path might be wrong.
TypeError: a bytes-like object is required. not 'str'. It's saying the function you called needs bytes-like input but you gave str, meaning I want binary data not text. So I thought, what data types are called bytes-like? Meaning anything that can behave like bytes, meaning mutable, like instead of "hello", b"hello".
What is each character in str that didn't have byte behavior? This question created for me, and with a bit of search, I found each character in str is a Unicode, meaning each character in international standard has a specific code called Unicode, and the goal was for a symbol or number to have unity. Each character. And now we reach that a str is a string of Unicode, so its type is not raw binary data. Bytes is a string of real bytes in Unicode characters. Each byte is a number between 0, 255 and has no character meaning, just raw data.
In the end, searching how to convert them to each other. For converting unicode to binary in file or network with attribute > encode, and for binary > str -> decode. This difference in attributes gave me a new view for errors, to keep in mind when such errors happen. When using decode attribute for str -> attributeerror we get.
I used b"hello" in this test according to my searches. Question came up, what's the difference between encode, b"---"? With searching, I reached that b"hello" directly makes a byte object, but encode is for converting string to binary.
In this meantime, I hit the word literal that I didn't know exactly in programming. With searching, I reached that literal means a fixed value directly written in code and itself and its type specified. Then I said to myself, its type is specified, what else? Then I understood every data you write every moment Python knows what type it is and no need for conversion. Like x = 52, 52 is literal. In my mind, I searched for non-literal and didn't reach a result, searched and found that input is non-literal. I got a bit confused, said to myself that's specified what its default data type is. Then I didn't notice this point that it's not specified what will be written.
Following this discussion, this point in test writing, never replace all opens with mock_open, read_data, because some libraries expect bytes. When we use builtins.open, patch in test, all opens in test environment use this path. This can cause unexpected behavior.
Notes: Pay attention to how opens are opened. Example mode='r', mode='rb'. In this meantime, if we have multiple opens that differ in mode and we have builtins.open, this errors might come again. And only mock the ones I really want because this open might need access to real object.
error: recursionError: maximum recursion depth exceeded. Recursion means return, and in this error it says the function is stuck in itself, and with traceback you can see which function. And this error in Python default after 1000 tries.
error: most likely due to a circular import , partially initialize error. Meaning Python couldn't import what I want. Meaning the file you imported itself at the same time imports the file you're in. Example module b tries to import func a from module a, but module a not fully initialized yet, so called partially initialize Python says this.
Solutions for this error several. We can put import in the function we want to use. This method called lazyimport but when executing we need to check project structure well because with lots of execution of that function each time import happens and this causes overhead -> memory, space, .. There's a way to change order or structure of imports. And last way that's dangerous is using patch with importlib. If no other method works, use this.
Question for me that we import, why for example that file even though imported but not created and not initialized. In reality with a bit search on this I found that when importing class, function, and variables not created. Why? Because when Python imports a module, Python makes a temp dictionary for it and they only created when executed.
I reached this that with Debug you can find problem from which import but how is debug style for this error. I understood I need to pay attention to modules and see what modules are imported from it. I tried to see if can't put log something like print. Ah with this error files don't run at all until I reached a solution. Need to reach traceback error. What is traceback? Track of program execution meaning what error in what line happened. I need to put print, logger above each module import this way I understand up to where run and in which import hit problem.
In the end, I reached a Python design point that better to keep dependencies always one way and modules shouldn't be circular dependent. In these searches and searches it was written don't patch module well first this sentence a bit vague for me until with thought and check on sentence I understood means importlib or same last way and said without check don't do. Reason was yes causes load module but might sometimes from uninitialized variables for this might attributeerror, crash program.
TypeError: 'mock' object is not subscriptable. I myself in this case this error this way think that function I used mock in didn't fulfill its want mock. This while I still didn't understand the error. This is saying that objct here mock doesn't have the capability you used for it. What capability? getitem doesn't have to be like a dict this attribute to have and behave meaning presupposition dict not and you're using it as a dict.
'obj' object is not subscriptable. The line I misunderstood why? With wrong tone I read it and this caused me to guess what it says not to understand really what it's saying. This reason was I didn't understand error in first view. Mock from beginning doesn't have indexing capability that's why this error comes. Better in such times use MagicMock that has this capability but MagicMock sometimes dangerous acts.
We can solve this error for mock this way that return_value=dict() give it or mock.getitem.return_value = ... But in my investigations about it I found with MagicMock cleaner.
Overall mistakes I had so far go back to my rushing in thinking and to solve it quick. Fast reading error doesn't let me think on it to understand what it says.
We can use spec, spec_set for mock too that give obj to it and makes like that obj have its needed attributes. mock(spec=example:dict()) ... mock(spec_set=example: dict())
Question came up for me difference spec, spec_set what is. With spec we can give any attribute to mock while maybe that obj doesn't have that attribute at all and not checked. But spec_set exactly checks exactly only and only have that obj's attributes.
Few things I read to understand difference mock, MagicMock written that MagicMock might be too permissive and flexible and hide api mistakes. Meaning MagicMock does whatever you want from it even if in obj that attribute not exist. Because mock has all Python magic methods len, getitem, iter, enter, exit even if the obj we're testing doesn't have such capabilities.
In the end mock better for precise tests. This way I read.
A question came up for me and that was what exactly are Python magic methods. They call them magic methods or Python magic methods then I saw a semantic encounter from them len(a) = a.len() exp['test'] = exp.getitem('test') for i in items : -> items.iter() enter, exit: obj.enter() try: .... finally: obj.exit() To such understanding about them.
Such things in MagicMock simulated and for this reason dangerous because might obj not have these capabilities but attributed to it.
Inputs of exit was question for me cause it's exit part of contextmanager or closes with error or runs. What were those variables exc_type = exception_type exc_val = exception_value exc_tb = exception_traceback Meaning its execution logic this way that contextmanager first calls enter then executes and finally: exit.
From one error see where I got pulled. If I knew better was: If I knew correct debug way much better until project not finished under each variable or print, logger put put see really called empty full and .... To me logger and debug most important part of life too can be. This where what happened to be in the know.
A bit about readme I studied. First my view about it was a bit about project say capabilities and how to use and few links and done. A bit curious became how to write a correct readme file. I understood writing a correct readme file how much view presenting my project effect had. Main that my project audience who is while at project start didn't pay attention at all. Developer ordinary user or devops companies. Second what is my project feature. What important in it. What is my project essence performance simplicity or scalable. Caused me to understand need to express project details that during this path reach strengths and weaknesses of project.
readme not a simple intro and shouldn't be basically identity and ID card of project. For having good readme need to best knowledge of project you're working on reach. Well obvious if I can't know it well obviously others can't. I according to these things now read sure that in quality of my next project with better cognitive view act myself I thought that if in readme write that I have 100% coverage very bottom but need to pay attention even if truth express for document have and show it.
In the end up to here this project I did and finished it and its functionality correct. I don't feel satisfaction cause few non-functional flaws had. First convinced myself it's simple mvp no need to fix those flaws. Asked myself does project work correct? Answer yes was. Its functionality really correct was. But difficulty in setup had. This a bit engaged my mind. Some other limitations too had that could overlook for presenting code. But my project still hadn't found essence for it. And that in running tests a limitation existed that must before project config run tests and if if you wanted after config run tests a bit tests with produced files conflict.
This reason my frustration from project was. And hard was to fix it not technically but spiritually. Other everything for me monotonous became reason monotonous this was that first project did then tested it. Other from mock, patch, MagicMock in a row tired became and other no tolerance had. Then myself with thousand ways convinced that no dude no problem upload it don't take hard cause before reached this that don't take hard. Even in LinkedIn and Twitter I talked about this. But in the end if want one thing to myself remind is that taking hard with having essence two different things don't mistake these. Need to code in a way that signature of my way of thinking on that project be.
Top comments (0)