DEV Community: ANSHUL VAIBHAV

S3 Select in 2026: Why New Accounts Get MethodNotAllowed, and What to Use Instead

ANSHUL VAIBHAV — Fri, 06 Nov 2026 05:58:47 +0000

SELECT * FROM S3Object s LIMIT 5

That was the query I expected to work. I had uploaded a tiny CSV to Amazon S3, checked its metadata, and previewed the contents. When I clicked Run Query in Buckets by Serverless Creed, I got this instead:

The specified method is not allowed against this resource.

Changing the SQL didn't resolve it. Running the equivalent operation through AWS CLI returned MethodNotAllowed too.

This post follows that investigation across four scenarios in Buckets v2.8.7 and v2.8.9. It also explains the S3 Select availability change behind a plausible diagnosis, and the alternatives I would consider for a new project in 2026.

What S3 Select was designed to do

S3 Select lets an application send a SQL expression against one S3 object and receive selected records instead of downloading the entire object first.

It supports a subset of SQL over CSV, JSON and Apache Parquet inputs, with CSV or JSON output. For example, an application could request particular employee fields or filter records by department.

The single-object scope matters. A query against S3Object isn't a query across a bucket or a database of related tables.

What changed on July 25, 2024

In its July 2024 announcement, AWS closed new customer access to S3 Select, effective July 25, 2024. Existing eligible customers can continue using it. AWS also stated that it did not plan to introduce new capabilities.

The distinction is eligibility to use S3 Select, rather than simply how old an AWS account is. An older account isn't automatically evidence of existing S3 Select access.

My observed MethodNotAllowed response is consistent with these availability restrictions. However, AWS Support did not independently confirm the test account's eligibility, and the error alone does not establish the exact cause.

That qualification matters throughout the tests below.

The CSV query in Buckets — TC-S3SEL-001

I created a CSV with six employee records and a header row. It was 166 bytes, uncompressed, and encrypted with SSE-S3 (AES256).

The upload, metadata verification and object preview succeeded. In the query dialog, I chose CSV input, None for compression, and CSV output, then submitted the query at the start of this post.

Detail crops from the same testing screenshot: the CSV settings and failure notification.

Buckets displayed the failure rather than query results. Being able to upload and preview the object established that those workflows worked; it did not establish that S3 Select was available.

SSE-S3 also wasn't inherently an incompatible choice: the SelectObjectContent API documentation supports server-side encrypted objects, including SSE-S3.

Reproducing it outside Buckets

I next used AWS CLI to separate the application interface from the underlying service operation. This is the equivalent command with generic resource names:

aws s3api select-object-content \
  --bucket example-test-bucket \
  --key data/report.csv \
  --expression "SELECT * FROM S3Object s LIMIT 5" \
  --expression-type SQL \
  --input-serialization \
    '{"CSV":{"FileHeaderInfo":"USE"},"CompressionType":"NONE"}' \
  --output-serialization '{"CSV":{}}' \
  --region us-east-1 \
  /tmp/select-results.csv

FileHeaderInfo: USE tells S3 Select to treat the first CSV line as a header. The operation and serialization options are documented in the AWS CLI reference.

My independent CLI attempt returned:

An error occurred (MethodNotAllowed) when calling the
SelectObjectContent operation: The specified method is
not allowed against this resource.

The independent CLI attempt returned the same error. Private resource and profile details are covered.

This was the strongest evidence in the investigation: the failure was reproducible without Buckets. It narrowed the investigation, while leaving the exact AWS-side reason unresolved.

Invalid SQL didn't produce a syntax error — TC-S3SEL-002

The next scenario was intended to check SQL error handling. I deliberately submitted:

SELECT FROM S3Object s

I received the same MethodNotAllowed response. Correcting the expression and retrying didn't change the outcome.

Invalid SQL produced the same service response as the corrected query.

The editor remained responsive and let me retry. But I did not verify how Buckets presents an actual SQL syntax error: no parser-specific response was observed. The test also cannot reveal AWS's internal order of request validation.

JSON reached the same boundary — TC-S3SEL-003

I then created an uncompressed JSON Lines file with five employee records, totaling 314 bytes. Buckets identified it as JSON. With JSON input and CSV output selected, I submitted:

SELECT s.name, s.department
FROM S3Object s
WHERE s.department = 'Engineering'

The JSON query also returned MethodNotAllowed; no matching records were returned.

The expression was intended to select names and departments for Engineering employees. That describes the query's intention, rather than an observed result. AWS rejected this request with the same error as the CSV request.

A clearer explanation in Buckets v2.8.9

During the later JSON testing, v2.8.9 gave a more useful explanation of the service restriction. It discussed possible S3 Select availability issues and retained the original AWS error code, response and request identifiers.

The explanation was careful about uncertainty: possible causes included account eligibility and unsupported resource types. The AWS API reference confirms that directory buckets and S3 on Outposts do not support this operation. Those are documented possibilities, not confirmed findings about my test environment.

The expanded response is visible in the BZIP2 screenshot below, with request identifiers covered. It helped explain why repeatedly rewriting the SQL might not resolve this particular failure.

What the compression controls proved — TC-S3SEL-005

Buckets offered None, GZIP and BZIP2 in its compression dropdown. I could select GZIP and BZIP2 successfully.

GZIP was selectable. Selecting an option does not compress the stored object.

I reused the existing uncompressed JSON object and selected BZIP2. No compression-specific mismatch warning appeared before submission. When I ran the query, AWS returned MethodNotAllowed again.

BZIP2 selected for an uncompressed object. The displayed explanation concerns S3 Select availability, not a detected compression mismatch.

The dropdown worked as a UI control. Actual AWS decompression and compression-mismatch handling were not verified. I cannot claim that Buckets detected the mismatch.

AWS's format requirements distinguish these cases:

Input	Supported compression
CSV or JSON	Uncompressed, GZIP or BZIP2
Parquet	Columnar GZIP or Snappy; no whole-object compression

S3 Select also has a 256 KB SQL-expression limit and a 1 MB input/result-record limit. These constraints are separate from service eligibility. Proper compression testing would need working S3 Select access and correctly compressed fixtures as well as mismatch cases.

Separating service behavior from application behavior

These tests made the responsibility boundary clearer:

Responsibility	AWS	Buckets
Request setup	Defines the API and supported formats	Collects SQL and input/output options
Execution	Evaluates the request and returns data or an error	Submits the configured request
Diagnostics	Returns the service code and response	Displays the response and explains possible causes

The CLI reproduction supports treating this observed failure as an AWS service response rather than assuming a defect in the Buckets SQL editor. It doesn't certify every part of the application: successful execution, syntax-error presentation and compression processing still need verification in an eligible environment.

Amazon Athena as an alternative

For SQL over a dataset spanning many S3 objects, Amazon Athena is a more suitable direction. It is a serverless analytics service that queries S3 data using SQL, with broader analytical capabilities than S3 Select's single-object API.

In a common setup, the AWS Glue Data Catalog stores table metadata: column names and types, data locations, format information and partitions. A crawler can discover that metadata, or it can be defined manually. A crawler is optional; see the Glue catalog documentation.

Illustrative architecture. Glue supplies metadata; it is not an intermediate store for the source records.

After defining a suitable employee table, the equivalent analytical query could be:

SELECT name, department
FROM employees
WHERE department = 'Engineering';

This is an illustrative Athena query, not something successfully executed during these S3 Select tests. Moving to Athena requires table definitions, permissions and result configuration; it isn't just changing an API name.

Where the results go

With customer-managed results, Athena writes to an S3 location selected by the client or workgroup. An enforced workgroup setting takes precedence. For API calls, StartQueryExecution.OutputLocation supplies the client-side location; a console setting alone does not configure API calls. See result-location settings.

Alternatively, a workgroup can enable Athena managed query results. This removes the need to supply your own results bucket. Results are retained for 24 hours at no additional managed-results storage cost, then automatically deleted. This mode does not support query-result reuse.

Costs, limits and when I would process locally

AWS currently illustrates on-demand Athena SQL pricing at US$5 per TB scanned on its pricing page. Check the applicable Region and current rate before estimating a workload. Billing concerns data scanned, rather than just the rows returned. AWS documents rounding to the next MB and a 10 MB minimum per query in its cost-model guidance.

Other models include SQL Capacity Reservations, billed for compute capacity, and Apache Spark, billed for compute use. S3 storage, requests and applicable transfer charges remain relevant, including customer-managed results. Glue catalog usage can incur charges beyond its free allowance; optional crawlers and ETL jobs are billed separately under Glue pricing.

For larger datasets, partitioning, compression and columnar formats such as Parquet can reduce scans. Workgroup scan controls can help constrain query usage.

Athena still has service quotas, including regional concurrency and query timeouts, and a 262,144-byte query-string limit. Its SQL limitations include a 32 MB row/column size ceiling.

For my 166-byte CSV or 314-byte JSON fixture, I would usually download the object and filter it with Python, pandas or DuckDB. That avoids catalog setup for a tiny one-off task, though S3 requests, transfer and local resources still matter. Athena becomes more compelling when the dataset and repeated analytical work justify that setup.

What I learned

No scenario produced successful S3 Select query results. The useful findings were the independent CLI reproduction, the working input controls, and the clearer explanation in v2.8.9.

The lesson I will carry forward is to reproduce an unexpected service error outside the application before deciding which layer needs fixing. Here, that prevented a blocked execution test from becoming an unsupported claim about SQL or compression behavior.

My next article will follow the Glue, Athena and S3 Tables workflow: defining queryable data, running SQL, inspecting result storage and understanding the costs. Those are the next steps to test, rather than outcomes established here.

References

A Deep Dive into RAG: From Basics to Better Retrieval

ANSHUL VAIBHAV — Tue, 27 Oct 2026 03:37:33 +0000

An LLM can explain a software feature quite well and still give you the wrong steps to use it.

Consider a support chatbot answering this question:

How can I recover a deleted file in version 3?

It might describe a menu from version 2, confuse the product with another application, or suggest a recovery option that does not exist. The answer can sound convincing even when the model has never seen the relevant manual.

This is the problem I wanted to understand while reading The Survey of Retrieval-Augmented Text Generation in Large Language Models by Yizheng Huang and Jimmy X. Huang. I used the August 2024 version.

Retrieval-Augmented Generation (RAG) combines a search step with an LLM that writes the answer. The paper is a survey of existing research. What I found useful was how it breaks RAG into smaller decisions: how to prepare information, find it, select it, and use it in an answer.

I'll use a hypothetical support chatbot throughout this post. It searches product manuals, troubleshooting documents, and release notes.

What happens without RAG?

Without external retrieval, an LLM answers using what it learned during training and whatever information we include in the conversation.

That is often enough for a general explanation. It becomes less dependable when the question needs private documentation or details from a recent release.

We could paste the relevant manual into the prompt. For a small amount of information, that may be all we need. But someone still has to find the right manual and the right section. Doing that for every question becomes inconvenient as the document collection grows.

RAG automates that step. The application retrieves relevant information and gives it to the model before asking it to answer.

For our chatbot, the benefit is straightforward: it can consult the version 3 documentation instead of relying only on the model's learned knowledge. Updating the documents also does not require retraining the generator each time.

This does not guarantee a correct answer. If retrieval finds the wrong procedure, the model can still explain the wrong procedure very clearly.

The basic RAG workflow

The survey starts with three stages:

Indexing → Retrieval → Generation

Indexing prepares documents for searching. A long manual is usually split into smaller passages, or chunks. The application builds searchable indexes over these passages.

For keyword search, the index helps locate terms. For vector search, an embedding model turns passages into numerical representations that can be compared with a question.

Retrieval happens when the user asks something. The application searches for passages likely to contain the answer.

Generation comes afterward. The LLM receives the question and selected passage text, then writes the response. Embeddings locate the text; the generator reads the text.

Document preparation runs when documents are added or changed. It does not need to run again for every question.

Figure 2, page 3 of Huang and Huang's survey. Cropped from the paper. The upper part shows basic RAG; the lower part groups the areas where it can be improved.

How the basic workflow can be improved

Traditional RAG follows a straightforward process. Documents are split into chunks and indexed. When a question arrives, the application searches for relevant passages and sends the selected text, together with the question, to the LLM. The model then generates an answer. This basic setup often uses one retrieval pass, so its answer depends heavily on what that first search finds.

Enhanced RAG adds ways to improve the evidence before it reaches the generator. Documents can be cleaned, summarized, or enriched with useful context before indexing. The question can be expanded or reformulated to help search. Retrieved passages can then be filtered and re-ranked. If the evidence is still insufficient, the application can refine the question and search again. Once enough supporting information is available, the LLM combines it into an answer that can be tailored to the reader.

That is the broader workflow illustrated in the paper's detailed diagram below: prepare the data, refine the query, retrieve, check the results, and generate. The survey groups these ideas into pre-retrieval, retrieval, post-retrieval, and generation. Here, “enhanced RAG” describes a set of possible improvements, not one fixed model or a requirement to use every step. For example, hybrid search can also be part of a simple pipeline.

For our support chatbot, these changes address concrete problems: a chunk missing its version number, a question phrased differently from the manual, or a search that finds the recovery steps but misses a separate retention policy.

1. Give each chunk enough context

A chunk containing “This option is available for 30 days” is not very useful by itself. Which option? For which product?

For our chatbot, I would try keeping the section heading, recovery steps, and associated limitations together. Metadata such as the product version and document title can provide additional context.

Small chunks can lose meaning; large ones can include unrelated material. The right size depends on the documents.

Before changing models, I would read a few chunks and check whether they still make sense.

2. Make the question easier to search

Users do not always use the terminology found in documentation. Query manipulation helps bridge that gap.

The survey distinguishes three ideas:

Reformulation: rewrite “get my file back in v3” as “recover a deleted file in version 3.”
Expansion: add related terms such as “restore” or “removed document.”
Normalization: standardize wording such as “v3” and “version 3.”

The rewrite must preserve the user's intent. Dropping the version number might retrieve more results while making them less useful.

I would add this step when test questions reveal a wording problem, rather than assume every question needs an extra LLM call.

3. Use keyword and semantic search together

BM25 ranks passages using indexed terms. It considers factors such as term frequency, how common a term is across documents, and document length. It is not whole-sentence exact matching: depending on the query settings, matching some terms can be enough to return a result. Elastic explains the mechanics here.

Dense retrieval compares learned representations of the question and passages. It can help connect “recover a removed file” with “restore a deleted document,” although it can still miss relevant evidence.

Hybrid search combines these approaches. In a support application, lexical search can help with a specific error code, while semantic search can help with a user's description of the problem.

The results are combined into a ranked list. Whether this improves our chatbot is something to test against a simpler baseline.

4. Re-rank and filter what comes back

The first search finds candidates efficiently. A re-ranker takes a closer look at those candidates and changes their order.

A cross-encoder, for example, processes a question and passage together to score their relevance. Doing this across a large collection is expensive, so it is usually applied to a smaller retrieved set. The Sentence Transformers documentation illustrates this separation.

For our chatbot, re-ranking could move the actual recovery procedure above a passage that only mentions deleted files.

Filtering removes unsuitable material, such as duplicate passages or instructions explicitly limited to another version. Re-ranking changes priority; filtering changes what remains.

A BERT-based embedding retriever and a BERT-based cross-encoder have different jobs here. BERT itself is not a vector database or search engine.

5. Search again when something is missing

Suppose the manual says recovery depends on a separate retention policy. The first search has found part of the answer, but not enough to finish it.

An iterative approach can use that result to search for the policy before generating the final response. Section 4.2 of the survey discusses this along with recursive, conditional, and adaptive retrieval strategies.

Extra searches need a reason and a stopping condition. Otherwise, they add delay and cost without adding evidence.

Figure 4, page 10 of the survey, reproduced as a crop. The paper uses a climate-change example here. Its displayed scores are illustrative.

Follow the retrieval-quality decision near the bottom left: insufficient evidence leads back to query refinement; sufficient evidence proceeds toward generation.

This particular figure filters before re-ranking. That is one arrangement, not a universal order for all RAG systems.

6. Turn the evidence into a clear answer

The survey describes enhancing as combining and organizing retrieved information. Our chatbot might combine recovery steps from a manual with a limitation from the release notes.

Customization adapts the explanation to the reader. A beginner may need numbered steps; an experienced user may prefer a short answer. The underlying facts should remain the same.

If the documents do not establish a recovery method, the answer should say so rather than fill the gap with invented steps.

What is used in current systems?

These October 2026 documentation examples show implemented capabilities, rather than industry-wide adoption figures.

Hybrid retrieval and re-ranking: Azure AI Search documents a classic RAG approach using hybrid search and semantic ranking. Amazon Bedrock also supports reranker models with Knowledge Bases. These techniques are available as managed features, not only as research implementations.

Contextual retrieval: Anthropic's approach adds a short, chunk-specific explanation before creating embeddings and a BM25 index. For our example, that explanation could clarify which product version a passage describes. This method was published in September 2024, after the survey version used here.

Agentic retrieval: Azure's documented approach can use an LLM to break a complex question into focused searches, run them in parallel, and combine the results. Its LLM-based query-planning capability is currently marked preview. It can help with harder questions, but adds work and latency compared with a single-query pipeline.

For this support chatbot, I would start by testing good chunks, version metadata, hybrid retrieval, and re-ranking. I would add query rewriting or repeat retrieval when failures show why they are needed.

How would I check whether it improved?

The survey's evaluation discussion gives three useful questions:

Context relevance: Did we retrieve useful evidence?
Answer relevance: Did the response address the question?
Faithfulness: Are the answer's claims supported by the retrieved material?

Factual correctness still matters. An answer can faithfully repeat an outdated manual and be wrong for the user's situation.

The main lesson I took from the survey is to inspect where the failure happened. If the right passage never reached the generator, a larger generator is not the first thing I would try.

Primary reference: Yizheng Huang and Jimmy X. Huang, The Survey of Retrieval-Augmented Text Generation in Large Language Models, arXiv:2404.10981v2, 23 August 2024. Figures 2 and 4 belong to the paper's authors.

Working with S3 Object Tags in Buckets: SET, MERGE and CLEAR

ANSHUL VAIBHAV — Sat, 17 Oct 2026 17:29:41 +0000

A practical walkthrough for keeping Amazon S3 objects organized without changing what is inside the files.

Object tags are easy to overlook when you are busy uploading and moving files around S3. But once a bucket starts filling up, a few consistent labels can make objects easier to organize and manage. The part that matters is how you update them: replacing a tag set is not the same as adding a tag to what is already there.

I used Buckets by Serverless Creed to explore three useful actions—SET, MERGE and CLEAR—and checked the results in the AWS S3 console. I also compared the downloaded files with their original SHA-256 hashes to make sure the contents stayed exactly the same.

A quick look at object tags

An S3 object tag is a key-value pair attached to an object. For example, env=test, team=intern or phase=review can describe where a file belongs or how it is used. These are separate from the file itself and from ordinary user-defined object metadata. S3 allows up to ten tags per object.

Tags can be useful for organizing data and for AWS features that are configured to use them, including lifecycle rules and some access-control policies. That is why updating tags deserves a little more care than simply changing a filename.

Start with a small sample folder

I worked with a handful of harmless text files in a private S3 bucket:

  docs/readme.txt
  b.txt
  c.txt
  control.txt

The files were small enough to inspect easily. The bulk operations shown below target only b.txt and c.txt. I left control.txt with a separate control=keep tag so there was always an object to compare against the ones being updated.

The sample files as they appear in the Buckets object browser.

SET: replace the existing tags

SET is the option to choose when an object needs a completely new tag set. In Buckets, select the files, open the bulk tagging action and choose Set (replace). For this example, I entered team=intern.

The important word is replace: tags that were on the object before, but are not included in the new set, will be removed. Before applying the change, the dialog shows the exact object keys being updated. It is worth reading that list rather than assuming every visible folder or file is included.

SET preview in Buckets, with the new team tag and the exact object list.

The operation completes and Buckets displays the update result.

Buckets reported Updated: 2, Failed: 0 for SET.

I then opened the object details in AWS. The tag panel showed one tag, team=intern, which is the state I expected after a replacement.

AWS S3 shows the new tag set after SET.

MERGE: add a tag without losing the others

MERGE is more useful when the existing labels should stay. I kept team=intern, switched to Merge, and entered a new tag: phase=review. I left the remove-keys field empty.

That should leave both tags on the objects. If a tag key already exists, MERGE updates that key; keys not mentioned are preserved. This is usually a safer option for incremental changes than rebuilding the full tag list by hand.

MERGE adds phase=review while preserving the other tag keys.

The MERGE action completes successfully.

Buckets reported Updated: 2, Failed: 0 for MERGE.

The AWS object details now displayed both team=intern and phase=review. Checking the entire tag list is useful here: seeing only the new key would not prove that the earlier one survived.

Both team and phase are visible in the AWS object-tag panel.

CLEAR: remove tags, not the object

CLEAR does exactly what the name suggests: it removes the object’s tag set. In Buckets, choose Clear all after selecting the intended files. The confirmation makes the scope explicit and warns that tag-dependent lifecycle rules or policies may no longer match.

This is the step where I would be most careful in a real bucket. Clearing tags does not delete a file, but tags may be used by other AWS configurations. Always check the objects listed in the confirmation before proceeding.

CLEAR warns about removing all tags and shows the affected object keys.

Buckets confirms that the tag-removal operation completed.

Buckets reported Updated: 2, Failed: 0 for CLEAR.

Afterward, the AWS object details showed Tags (0). The AWS object details still identified b.txt; the wider Buckets capture also shows readme.txt remaining in the browser after tag removal. That distinction is important: removing tags is not the same thing as deleting objects.

The S3 tag section is empty after CLEAR.

A wider view of the Buckets object browser after tag removal.

The AWS object details page still shows the file and its empty tag set.

Independent verification and SHA-256 integrity checking

I kept control.txt out of the tagging changes. Its control=keep tag was still there at the end, which was a quick check that the untouched object had retained its original tag.

The control object keeps its original tag.

For file contents, I recorded SHA-256 hashes of the original text files before uploading them. After the tagging sequence, I downloaded the S3 objects into a separate folder and compared them with that original hash list. Run the check from the download folder, with the original hash list one directory above it:

shasum -a 256 -c ../original-hashes.txt

All four downloaded files match their original SHA-256 hashes.

The output was:

docs/readme.txt: OK
b.txt: OK
c.txt: OK
control.txt: OK

The AWS Console screenshot lists CRC64NVME as its stored checksum function. The SHA-256 verification here is a separate local comparison of the downloaded bytes against the original files.

All four checks returned OK. That gives two independent views of the outcome: the AWS Console shows how the tags changed, and the hash check shows that the downloaded file bytes matched the originals.

A few habits worth keeping

Three things made this workflow straightforward. First, use SET only when replacing the full tag set is intentional; use MERGE when existing labels should remain. Second, review the exact object list before any bulk operation, especially when folders and nested files are involved. Third, check the resulting tags in S3 rather than relying on a success message alone.

For a small example like this, the difference between SET, MERGE and CLEAR is easy to see. On a busy bucket, choosing the right operation—and verifying the final state—can save a lot of confusion later.