sgx_scrapper/adv-investigation/README.md

# Advanced Investigation

This folder holds the detective work we did to figure out how SGX's website talks to its own data APIs.

## Why this exists

The SGX website does not publish a clean, public API manual. Instead, it has a list of API endpoints buried in a config file. Our first attempt (`Doc/` — previously called `discovery/`) simply knocked on every endpoint door and wrote down which ones opened and which ones did not. That gave us a rough map, but many doors were locked.

This folder contains the follow-up work: we opened a real web browser, visited the SGX site, and watched exactly which API calls the site makes when you click around. By copying what the website itself does, we found the missing keys: the right URL paths, the right settings, the right headers, and the secret token some APIs demand.

## What you will find here

- `investigation.md` — the full detective report. It lists, page by page, what we saw in the browser and what it taught us about each API.
- `payloads/` — the raw responses we captured from the website. These are the proof behind the report.
- `README.md` — this file.

## Simple summary of what we learned

1. **The website starts by loading a config file.** Before asking for stock prices or announcements, the SGX site fetches `https://www.sgx.com/config/appconfig.json?v=...`. That file contains a version number (`v`) and a CMS version hash. If the version changes, the API calls may change too, so it is worth checking this file first.
2. **Some APIs need extra information.** A bare API URL often fails. The site always adds a path, filters, or a `params` list to say exactly which fields it wants back.
3. **Some APIs need a token.** The announcements API, for example, will not talk to you unless you send a special header called `authorizationtoken`. We traced where the website gets that token: it asks a CMS endpoint called `we_chat_qr_validator`, receives a scrambled string, and unscrambles it with ROT13.
4. **CORS matters.** The APIs check where the request came from. You must send `Origin` and `Referer` headers that look like you are calling from `https://www.sgx.com` or `https://investors.sgx.com`.

## Tools we used

- **Chrome + chrome-devtools-mcp:** we browsed the real SGX website and recorded the network traffic.
- **`../scripts/inspect_cache.py`:** a helper that reads the saved API responses and prints a quick summary of what is inside.
- **Small Python snippets:** used on the fly to decode tokens, search JavaScript bundles, and double-check headers.

## Relationship to the rest of the project

- **`Doc/`** — the simplified API field guide. It tells you which endpoints work and how to call them. We are not going to re-run the original discovery sweep; this folder is now the maintained reference.
- **`adv-investigation/`** — the behind-the-scenes notes on how we figured those calls out.








# Working endpoint patterns (from browser recon, 2026-06-25)

Full capture logs and response bodies are in `adv-investigation/investigation.md` and `adv-investigation/payloads/`.

### Required headers for API replay

Most `api.sgx.com`, `api2.sgx.com`, and `api3.sgx.com` calls require:

```
Origin: https://www.sgx.com
Referer: https://www.sgx.com/
```

`api.sgx.com/announcements/v1.1/*` additionally requires an `authorizationtoken` header. The token observed in the current frontend bundle is documented in `adv-investigation/investigation.md`; it should be treated as a credential and not committed in plain text.

### Field projection via `params`

Securities, derivatives, and indices endpoints accept a `params` query string of comma-separated field abbreviations. Examples of abbreviations seen in browser traffic:

- Securities: `nc`, `n`, `type`, `ls`, `m`, `sc`, `bl`, `sip`, `ex`, `ej`, `clo`, `cr`, `cur`, `el`, `r`, `i`, `cc`, `ig`, `lf`, `adjusted-vwap`, `bond_accrued_interest`, `bond_clean_price`, `bond_dirty_price`, `bond_date`, `b`, `bv`, `p`, `c`, `change_vs_pc`, `change_vs_pc_percentage`, `cx`, `cn`, `dp`, `dpc`, `du`, `ed`, `fn`, `h`, `iiv`, `iopv`, `lt`, `l`, `o`, `p_`, `pv`, `ptd`, `s`, `sv`, `trading_time`, `v_`, `v`, `vl`, `vwap`, `vwap-currency`.
- Indices: `lp`, `trading_time`, `vl`.
- Derivatives: `base-date`, `total-volume`, `last-traded-price-abs`.

Omitting `params` may return an error or a different response shape.

### Securities (`SECURITIES_API_URL` = `https://api.sgx.com/securities/v1.1`)

| Call | URL pattern | Purpose |
|------|-------------|---------|
| Market aggregate | `.../aggregate/` | Total volume/value and advancer/decliner counts. |
| Volume by type | `.../aggregate/volume?securitytype=stocks` | Total stock volume. |
| Value by type | `.../aggregate/value?securitytype=stocks` | Total stock value. |
| Sector heat map | `.../sectorbreakdown` | 20 SSIC sectors with change/changePercentage. |
| Circuit breakers | `.../circuitbreaker` | Active circuit-breaker records. |
| Top advancers | `.../advancers/{stocks,reits,etfs,structuredwarrants,dlcertificates}/{N}/v?params=...` | Top N by value. |
| ETF snapshot | `.../etfs?params=n,nc,cx,lt,c,p,vl,v,sc` | ETF screener price data. |
| Full screener | `...?excludetypes=bonds&pagestart=0&pagesize=500&sort=vl&sortdir=desc&params=...` | Paginated, sorted securities table. |
| Reference snapshot | `...?params=nc,n,type,ls,m,sc,bl,sip,ex,ej,clo,cr,cur,el,r,i,cc,ig,lf` | Static-ish reference data. |
| Full quote | `...?params=nc,adjusted-vwap,bond_accrued_interest,bond_clean_price,bond_dirty_price,bond_date,b,bv,p,c,change_vs_pc,change_vs_pc_percentage,cx,cn,dp,dpc,du,ed,fn,h,iiv,iopv,lt,l,o,p_,pv,ptd,s,sv,trading_time,v_,v,vl,vwap,vwap-currency` | Rich pricing/volume snapshot. |

### Derivatives (`DERIVATIVES_API_URL` = `https://api.sgx.com/derivatives/v1.0`)

| Call | URL pattern | Purpose |
|------|-------------|---------|
| Total volume | `...?aggregate=true&aggregate-option=total-volume` | Single aggregate number. |
| Futures by commodity | `...?groupby=cc&category=futures` | All futures grouped by commodity code. |
| History by symbol | `.../history/symbol/{symbol}?days=10d&category=futures&params=...` | 10-day OHLC-like history. |

### Indices (`INDICES_API_URL` = `https://api.sgx.com/indices/v1.0/`)

| Call | URL pattern | Purpose |
|------|-------------|---------|
| Master list | `...` | All 854 indices with current OHLC/change. |
| Metadata | `.../pid/{pid}/` | Last price and metadata for a single index. |
| Intraday chart | `.../charts/intraday/pid/{pid}/{period}?params=lp,trading_time,vl` | Intraday time-series. |
| Historic chart | `.../charts/historic/pid/{pid}/{period}?params=lp,trading_time,vl` | Daily historic time-series. |

Periods seen: `5d`, `1m`. PIDs seen: `.STI`, `SGN50N`, `.SREIT`, `SGLC40CN`, `.APACFINN`, `.EMASIAUN`, `ASIATECN`, `IEVNTR`, `.FTFSTM`, `.FTFSTS`.

### Third-party indices (`THIRDPARTY_INDICES_API_URL` = `https://api.sgx.com/thirdpartyindices/v1.0/`)

| Call | URL pattern | Purpose |
|------|-------------|---------|
| ChinaBond | `.../chinabonds` | ChinaBond index data. |

### Announcements (`ANNOUNCEMENTS_API_URL` = `https://api.sgx.com/announcements/v1.1/`)

| Call | URL pattern | Purpose |
|------|-------------|---------|
| Issuer filter | `.../companylist` | All issuing companies. |
| Security filter | `.../securitylist` | All securities. |
| Count | `.../count?periodstart={ts}&periodend={ts}` | Count in date window. |
| List | `.../?periodstart={ts}&periodend={ts}&pagestart=0&pagesize=20` | Paginated announcement records. |
| Summary | `.../summary/company?periodstart={ts}&cat={cat}` | Category summary (`ANNC`, `CACT`, `PLST`, `TRAD`). |

### IPO (`IPO_PERFORMANCE_API_URL` and `IPO_PROSPECTUS_API_URL`)

| Call | URL pattern | Purpose |
|------|-------------|---------|
| Years | `https://api.sgx.com/ipoperformance/v1.0/yearlist` | Filter dropdown. |
| Issuers | `https://api.sgx.com/ipoperformance/v1.0/issuerlist` | Filter dropdown. |
| Companies | `https://api.sgx.com/ipoperformance/v1.0/companylist` | Filter dropdown. |
| Performance by year | `https://api.sgx.com/ipoperformance/v1.0?year=2026` | IPO performance data. |
| Prospectus | `https://api.sgx.com/ipoprospectus/v1.0/?closing_time={yyyyMMdd_HHmmss}` | Open/closed IPO issues. |

### ETF screener (`ETF_API_URL` = `https://api.sgx.com/etfscreener/v1.0`)

| Call | URL pattern | Purpose |
|------|-------------|---------|
| Documents | `.../documents` | ETF factsheet/document metadata. |

### CMS (`CMS_API_URL` = `https://api2.sgx.com/content-api`)

Calls take the form:

```
https://api2.sgx.com/content-api?queryId={CMS_VERSION}:{operation}&variables={urlencoded_json}
```

Current `CMS_VERSION` (from appconfig): `70f75ec90c030bab34d750ee55d74b016f70d4b6`.

Operations used by the frontend:

- `page` — page-level CMS content; variables include `path` and `lang`.
- `all_menus` — site navigation; variables: `lang`.
- `alerts` — global alert banners; variables: `lang`.
- `advertisement_list` — ads.
- `derivatives_products_list` — derivatives product list; variables: `limit`, `offset`, `lang`.
- `index_list` — index taxonomy; variables: `lang`.
- `market_updates_list` — market news list; variables: `category`, `limit`, `categoryFilterEnabled`, `lang`.
- `taxonomy_terms` — vocabulary terms; variables: `vid`, `lang`.
- `we_chat_qr_validator` — tiny validation call.

### Financial reports

`FINANCIAL_REPORTS_API_URL` should be called paginated:

```
https://api.sgx.com/financialreports/v1.0?pagestart={n}&pagesize=2000&params=id,companyName,documentDate,securityName,title,url
```

### Investor portal (`investors.sgx.com`)

The investor portal uses a different CMS `queryId` prefix (`9b5f40321735d88857c12bd08459b5aba7fe2d1e`) and a different `Origin`/`Referer`:

```
Origin: https://investors.sgx.com
Referer: https://investors.sgx.com/
```

Some calls (e.g. `snapshotreports`, `announcements/securitycode`) reuse the same `authorizationtoken` header used on `www.sgx.com`.

#### Per-stock quote and charts (`SECURITIES_API_URL`)

| Call | URL pattern | Purpose |
|------|-------------|---------|
| Stock details | `https://api.sgx.com/securities/v1.1/stocks/code/{code}?ts={timestamp}` | Per-stock quote/details. |
| Historic chart | `https://api.sgx.com/securities/v1.1//charts/historic/stocks/code/{code}/{1w,1m,1y,5y}?params=trading_time,vl,lt` | Historic OHLC-like time-series. |
| Intraday chart | `https://api.sgx.com/securities/v1.1//charts/intraday/stocks/code/{code}/1d?params=trading_time,vl,lt` | Intraday time-series. |

#### Fundamentals and quant data (v2.0 endpoints)

These endpoints are not listed in the original `appconfig.json` but are called by `investors.sgx.com`:

| Endpoint | URL pattern | Purpose |
|----------|-------------|---------|
| Stock screener | `https://api.sgx.com/stockscreener/v2.0/all?params=exchange,exchangeCountryCode,companyName,stockCode,marketCapitalization,salesTTM,priceToEarningsRatio,dividendYield,fourWeekPricePercentChange,thirteenWeekPricePercentChange,twentySixWeekPricePercentChange,fiftyTwoWeekPricePercentChange,netProfitMargin,returnOnAvgCommonEquity,priceToCashFlowPerShareRatio,totalDebtToTotalEquityRatio,salesPercentageChange,sector,priceToBookRatio,priceCurrCode` | Cross-sectional quant screener (~701 stocks) with market cap, P/E, yield, returns, sector, etc. |
| Analyst / valuation | `https://api.sgx.com/ratiosreports/v2.0/countryCode/SGP/stockCode/{code}?params=...` | Market cap, P/E, EPS, margins, ROA/ROE, debt/equity, dividend yield, beta, 52-week range, analyst consensus, target price. |
| Snapshot | `https://api.sgx.com/snapshotreports/v2.0/countryCode/SGP/stockCode/{code}?params=enterpriseValue,returnOnEquity,companyName,stockCode,sectorName,industryName,reportingCurrency,tradedCurrency,currentFiscalYear,currentFiscalYearEndMonth,currencyIdForMarketCap` | Enterprise value, ROE, sector/industry, reporting/traded currency. |
| Shareholders | `https://api.sgx.com/shareholdersreports/v2.0/stockCode/{code}?params=investorName,investorType,investorHoldingsDate,pctOfSharesOutstanding,sharesHeld,sharesHeldChange,turnoverRating` | Major shareholder holdings. |
| Balance sheet | `https://api.sgx.com/financialstatementreports/v2.0/balanceSheet/countryCode/SGP/stockCode/{code}?params=all` | Balance sheet. |
| Cash flow | `https://api.sgx.com/financialstatementreports/v2.0/cashFlow/countryCode/SGP/stockCode/{code}?params=all` | Cash flow statement. |
| Income statement | `https://api.sgx.com/financialstatementreports/v2.0/incomeStatement/countryCode/SGP/stockCode/{code}?params=all` | Income statement. |

#### Per-stock corporate actions and announcements

| Endpoint | URL pattern | Purpose |
|----------|-------------|---------|
| Corporate actions | `https://api.sgx.com/corporateactions/v1.0?pagesize=10&pagestart=0&ibmcode={ibmcode}&params=id,anncType,dateAnnc,exDate,name,particulars,recDate,datePaid&order=desc&orderBy=dateAnnc` | Corporate actions for a given `ibmcode`. |
| Announcements | `https://api.sgx.com/announcements/v1.1/securitycode?value={code}&cat=ANNC&securityCodeParams=securitycode&sub=ANNC17&pagestart=0&pagesize=250&periodstart={ts}` | Per-stock announcements filtered by security code. |

#### CMS document list

`https://api2.sgx.com/content-api?queryId=0c6b763562fa9ad6d4b40558d2670db60437f7a6:document_list&variables={...}` is used to fetch document metadata such as the `IP-CA-TABLE` JSON (`https://api2.sgx.com/sites/default/files/2026-06/IP-CA-TABLE-20260619_1.json`).

#### Additional CMS operations on `www.sgx.com`

- `data_settlement_prices_list` — derivatives settlement prices; variables include `underlyingProductsCategory` and filter flags.
- `prices_list` — general prices document list.

#### Fixed-income / bond endpoints (`SECURITIES_API_URL`)

| Call | URL pattern | Purpose |
|------|-------------|---------|
| Bonds snapshot | `https://api.sgx.com/securities/v1.1/bonds?params=nc,adjusted-vwap,bond_accrued_interest,...,vwap-currency` | All bonds. |
| Retail bond detail | `https://api.sgx.com/securities/v1.1/retailbonds/code/{code}?params=...` | Single retail bond. |
| Retail preference share | `https://api.sgx.com/securities/v1.1/retailpreferenceshares/code/{code}?params=...` | Single retail preference share. |
| Bond chart | `https://api.sgx.com/securities/v1.1/charts/historic/retailbonds/code/{code}/{period}?params=...` | Historic bond chart. |

#### Legacy infofeed endpoints (`api3.sgx.com`)

The `V1_DERIVATIVES_DAILY_LIST_URL` and `V1_DERIVATIVES_HISTORICAL_LIST_URL` endpoints accept an additional `noCache={timestamp}` parameter:

```
https://api3.sgx.com/infofeed/Apps?A=COW_Infopubdtstat_Content&B=DailyDataDownload&S_T=1&C_T=60&noCache={timestamp}
https://api3.sgx.com/infofeed/Apps?A=COW_Tickdownload_Content&B=TimeSalesData&C_T=20&noCache={timestamp}
```

### Corporate Actions (`CORPORATE_ACTIONS_API_URL` = `https://api.sgx.com/corporateactions/v1.0`)

| Call | URL pattern | Purpose |
|------|-------------|---------|
| Filter metalist | `.../metalist` | `companyName` filter values for the UI dropdown. |
| Global list | `...?pagestart=0&pagesize=20&params=id,anncType,datePaid,exDate,name,particulars,recDate` | Paginated global corporate-actions feed. |
| Per-stock list | `...?pagesize=10&pagestart=0&ibmcode={ibmcode}&params=id,anncType,dateAnnc,exDate,name,particulars,recDate,datePaid&order=desc&orderBy=dateAnnc` | Corporate actions for a single IBM code. |

### Circulars (`CIRCULARS_API_URL` = `https://api.sgx.com/circulars/v1.0`)

| Call | URL pattern | Purpose |
|------|-------------|---------|
| Filter metalist | `.../metalist` | `companyName` filter values. |
| List | `...?pagestart=0&pagesize=5` | Prospectus and supplement/circular records (`id`, `documentDate`, `prospectusType`, `subject`, `companyName`). |

### Meeting Schedules (`MEETING_SCHEDULES_API_URL` = `https://api.sgx.com/meetingschedules/v1.0`)

| Call | URL pattern | Purpose |
|------|-------------|---------|
| Filter metalist | `.../metalist` | `issuerName`, `securityName`, `anncType` filter lists. |
| List | `...?pagestart=0&pagesize=20&params=issuerName,securityName,meetingDateTime,anncType,anncTitle,recDateTime` | Paginated AGM/EGM schedule records. |

### Corporate Information (`CORPORATE_INFORMATION_API_URL` = `https://api.sgx.com/corporateinformation/v1.0`)

| Call | URL pattern | Purpose |
|------|-------------|---------|
| Filter metalist | `.../metalist` | `issuerName`, `securityName`, `industry`, `country` filter lists. |
| List | `...?pagestart=0&pagesize=20&params=issuerName,securityName,industry,country,incorporationDate,listedDate,tradingCurrency,url,email` | Paginated issuer profile records. |

### Wholesale Bonds (`WHOLESALE_BONDS_API_URL` = `https://api.sgx.com/wholesalebonds/v1.0`)

| Call | URL pattern | Purpose |
|------|-------------|---------|
| Filter metalist | `.../metalist` | `issuerName`, `securityName` filter lists. |
| List | `...?pagestart=0&pagesize=20&params=issuerName,securityName,bondType,lastTradedPrice,lastTradedDate,lastTradedYield,couponRate,maturityDate,bondPriceType` | Paginated wholesale bond price/yield records. |

### Evaluated Bond Prices (`EVALUATED_BONDS_PRICES_API_URL` = `https://api.sgx.com/evaluatedbondprices/v1.0`)

| Call | URL pattern | Purpose |
|------|-------------|---------|
| Filter metalist | `.../metalist` | `issuerName`, `securityName` filter lists. |
| List | `...?pagestart=0&pagesize=20&params=issuerName,securityName,cleanPrice,dirtyPrice,evaluatedPriceDate,bidPrice,offerPrice,yield,couponRate,maturityDate` | Paginated evaluated bond prices. |

### Negotiated Large Trades (`NLT_API_URL` = `https://api.sgx.com/negotiatedlargetrades/v1.0`)

| Call | URL pattern | Purpose |
|------|-------------|---------|
| List | `...?pagestart=0&pagesize=20` | Paginated NLT records (`contractCode`, `instrumentGroup`, `category`, `month`, `year`, `strikePrice`, `quantity`, `businessDate`, `tradedDate`, `clearedOnDate`, `premium`, `session`). |

### Structured Warrants (`STRUCTURED_PRODUCTS_API_URL` = `https://api.sgx.com/marketmetadata/v2/structure-warrants`)

| Call | URL pattern | Purpose |
|------|-------------|---------|
| Metadata | `...?pagestart=0&pagesize=...` | Structured-warrant instrument metadata (1,183 records). |