Using tempfile module in Python

Senior Data Engineer • Contractor / Freelancer • GCP & AWS Certified
Search for a command to run...

Senior Data Engineer • Contractor / Freelancer • GCP & AWS Certified
No comments yet. Be the first to comment.
Concise guides to Python features and patterns most useful for data engineering — from built-ins to standard library modules.
If you ever need to work with JSON files in Python, you're going to encounter the module with the same name. It help encode to and decode from JSON. Here are the basics: ➡ json.load imports contents from a JSON file to a Python object, based on conve...
Here's a useful Dataform concept: pre_operations and post_operations. As the name implies, these represent a set of actions that run before and after the main operation (table, view, or SQL operations

BigQuery has always been a SQL engine for tabular data. Object tables add an interesting twist to that. Instead of rows containing values, an object table gives you one row per file — pointing at da

Query your data lake with warehouse-grade security and performance — without moving a single file.

Ever run a heavy BigQuery SQL query, processed gigabytes of data — and then accidentally closed the tab or forgot to save the results? 😬 Don't re-run it. Your results are still there. BigQuery automa

You can use query parameters in BigQuery hashtag#SQL (now in the console as well!) — but how are they different from variables, and when should you use each? Both parameters and variables act as place

In this world, everything is ephemeral. If you need to create temporary files or directories in Python, check out the tempfile module.
Whether you want to store intermediate results, manage temp data during execution or just test things out, it can help you by abstracting file creation and cleaning-up operations.
In the a quick walk-through below, we're looking at the following functions:
- TemporaryFile : creates an anonymous temporary file
- NameTemporaryFile: creates a named temp file which we can use in multiple contexts
- TemporaryDirectory: creates a temporary directory (in which we also can create temp files).
Notice how we're using the inside context managers (the 'with' block , check out the comments for a quick intro on them). This means the file will be automatically cleaned up upon exiting (default behavior), unless we specify delete=False at creation.
Found it useful? Subscribe to my Analytics newsletter at notjustsql.com.
Enjoyed this? Here are some related articles you might find useful: