sgx_scrapper/adv-investigation/README.md

Advanced Investigation

This folder holds the detective work we did to figure out how SGX's website talks to its own data APIs.

Why this exists

The SGX website does not publish a clean, public API manual. Instead, it has a list of API endpoints buried in a config file. Our first attempt (Doc/ — previously called discovery/) simply knocked on every endpoint door and wrote down which ones opened and which ones did not. That gave us a rough map, but many doors were locked.

This folder contains the follow-up work: we opened a real web browser, visited the SGX site, and watched exactly which API calls the site makes when you click around. By copying what the website itself does, we found the missing keys: the right URL paths, the right settings, the right headers, and the secret token some APIs demand.

What you will find here

• investigation.md — the full detective report. It lists, page by page, what we saw in the browser and what it taught us about each API.
• payloads/ — the raw responses we captured from the website. These are the proof behind the report.
• README.md — this file.

Simple summary of what we learned

1. The website starts by loading a config file. Before asking for stock prices or announcements, the SGX site fetches https://www.sgx.com/config/appconfig.json?v=.... That file contains a version number (v) and a CMS version hash. If the version changes, the API calls may change too, so it is worth checking this file first.
2. Some APIs need extra information. A bare API URL often fails. The site always adds a path, filters, or a params list to say exactly which fields it wants back.
3. Some APIs need a token. The announcements API, for example, will not talk to you unless you send a special header called authorizationtoken. We traced where the website gets that token: it asks a CMS endpoint called we_chat_qr_validator, receives a scrambled string, and unscrambles it with ROT13.
4. CORS matters. The APIs check where the request came from. You must send Origin and Referer headers that look like you are calling from https://www.sgx.com or https://investors.sgx.com.

Tools we used

• Chrome + chrome-devtools-mcp: we browsed the real SGX website and recorded the network traffic.
• ../scripts/inspect_cache.py: a helper that reads the saved API responses and prints a quick summary of what is inside.
• Small Python snippets: used on the fly to decode tokens, search JavaScript bundles, and double-check headers.

Relationship to the rest of the project

• Doc/ — the simplified API field guide. It tells you which endpoints work and how to call them. We are not going to re-run the original discovery sweep; this folder is now the maintained reference.
• adv-investigation/ — the behind-the-scenes notes on how we figured those calls out.

Working endpoint patterns (from browser recon, 2026-06-25)

Full capture logs and response bodies are in adv-investigation/investigation.md and adv-investigation/payloads/.

Required headers for API replay

Most api.sgx.com, api2.sgx.com, and api3.sgx.com calls require:

Origin: https://www.sgx.com
Referer: https://www.sgx.com/

api.sgx.com/announcements/v1.1/* additionally requires an authorizationtoken header. The token observed in the current frontend bundle is documented in adv-investigation/investigation.md; it should be treated as a credential and not committed in plain text.

Field projection via params

Securities, derivatives, and indices endpoints accept a params query string of comma-separated field abbreviations. Examples of abbreviations seen in browser traffic:

• Securities: nc, n, type, ls, m, sc, bl, sip, ex, ej, clo, cr, cur, el, r, i, cc, ig, lf, adjusted-vwap, bond_accrued_interest, bond_clean_price, bond_dirty_price, bond_date, b, bv, p, c, change_vs_pc, change_vs_pc_percentage, cx, cn, dp, dpc, du, ed, fn, h, iiv, iopv, lt, l, o, p_, pv, ptd, s, sv, trading_time, v_, v, vl, vwap, vwap-currency.
• Indices: lp, trading_time, vl.
• Derivatives: base-date, total-volume, last-traded-price-abs.

Omitting params may return an error or a different response shape.

Securities (SECURITIES_API_URL = https://api.sgx.com/securities/v1.1)

Call	URL pattern	Purpose
Market aggregate	.../aggregate/	Total volume/value and advancer/decliner counts.
Volume by type	.../aggregate/volume?securitytype=stocks	Total stock volume.
Value by type	.../aggregate/value?securitytype=stocks	Total stock value.
Sector heat map	.../sectorbreakdown	20 SSIC sectors with change/changePercentage.
Circuit breakers	.../circuitbreaker	Active circuit-breaker records.
Top advancers	.../advancers/{stocks,reits,etfs,structuredwarrants,dlcertificates}/{N}/v?params=...	Top N by value.
ETF snapshot	.../etfs?params=n,nc,cx,lt,c,p,vl,v,sc	ETF screener price data.
Full screener	...?excludetypes=bonds&pagestart=0&pagesize=500&sort=vl&sortdir=desc&params=...	Paginated, sorted securities table.
Reference snapshot	...?params=nc,n,type,ls,m,sc,bl,sip,ex,ej,clo,cr,cur,el,r,i,cc,ig,lf	Static-ish reference data.
Full quote	...?params=nc,adjusted-vwap,bond_accrued_interest,bond_clean_price,bond_dirty_price,bond_date,b,bv,p,c,change_vs_pc,change_vs_pc_percentage,cx,cn,dp,dpc,du,ed,fn,h,iiv,iopv,lt,l,o,p_,pv,ptd,s,sv,trading_time,v_,v,vl,vwap,vwap-currency	Rich pricing/volume snapshot.

Derivatives (DERIVATIVES_API_URL = https://api.sgx.com/derivatives/v1.0)

Call	URL pattern	Purpose
Total volume	...?aggregate=true&aggregate-option=total-volume	Single aggregate number.
Futures by commodity	...?groupby=cc&category=futures	All futures grouped by commodity code.
History by symbol	.../history/symbol/{symbol}?days=10d&category=futures&params=...	10-day OHLC-like history.

Indices (INDICES_API_URL = https://api.sgx.com/indices/v1.0/)

Call	URL pattern	Purpose
Master list	...	All 854 indices with current OHLC/change.
Metadata	.../pid/{pid}/	Last price and metadata for a single index.
Intraday chart	.../charts/intraday/pid/{pid}/{period}?params=lp,trading_time,vl	Intraday time-series.
Historic chart	.../charts/historic/pid/{pid}/{period}?params=lp,trading_time,vl	Daily historic time-series.

Periods seen: 5d, 1m. PIDs seen: .STI, SGN50N, .SREIT, SGLC40CN, .APACFINN, .EMASIAUN, ASIATECN, IEVNTR, .FTFSTM, .FTFSTS.

Third-party indices (THIRDPARTY_INDICES_API_URL = https://api.sgx.com/thirdpartyindices/v1.0/)

Call	URL pattern	Purpose
ChinaBond	.../chinabonds	ChinaBond index data.

Announcements (ANNOUNCEMENTS_API_URL = https://api.sgx.com/announcements/v1.1/)

Call	URL pattern	Purpose
Issuer filter	.../companylist	All issuing companies.
Security filter	.../securitylist	All securities.
Count	.../count?periodstart={ts}&periodend={ts}	Count in date window.
List	.../?periodstart={ts}&periodend={ts}&pagestart=0&pagesize=20	Paginated announcement records.
Summary	.../summary/company?periodstart={ts}&cat={cat}	Category summary (ANNC, CACT, PLST, TRAD).

IPO (IPO_PERFORMANCE_API_URL and IPO_PROSPECTUS_API_URL)

Call	URL pattern	Purpose
Years	https://api.sgx.com/ipoperformance/v1.0/yearlist	Filter dropdown.
Issuers	https://api.sgx.com/ipoperformance/v1.0/issuerlist	Filter dropdown.
Companies	https://api.sgx.com/ipoperformance/v1.0/companylist	Filter dropdown.
Performance by year	https://api.sgx.com/ipoperformance/v1.0?year=2026	IPO performance data.
Prospectus	https://api.sgx.com/ipoprospectus/v1.0/?closing_time={yyyyMMdd_HHmmss}	Open/closed IPO issues.

ETF screener (ETF_API_URL = https://api.sgx.com/etfscreener/v1.0)

Call	URL pattern	Purpose
Documents	.../documents	ETF factsheet/document metadata.

CMS (CMS_API_URL = https://api2.sgx.com/content-api)

Calls take the form:

https://api2.sgx.com/content-api?queryId={CMS_VERSION}:{operation}&variables={urlencoded_json}

Current CMS_VERSION (from appconfig): 70f75ec90c030bab34d750ee55d74b016f70d4b6.

Operations used by the frontend:

• page — page-level CMS content; variables include path and lang.
• all_menus — site navigation; variables: lang.
• alerts — global alert banners; variables: lang.
• advertisement_list — ads.
• derivatives_products_list — derivatives product list; variables: limit, offset, lang.
• index_list — index taxonomy; variables: lang.
• market_updates_list — market news list; variables: category, limit, categoryFilterEnabled, lang.
• taxonomy_terms — vocabulary terms; variables: vid, lang.
• we_chat_qr_validator — tiny validation call.

Financial reports

FINANCIAL_REPORTS_API_URL should be called paginated:

https://api.sgx.com/financialreports/v1.0?pagestart={n}&pagesize=2000&params=id,companyName,documentDate,securityName,title,url

Investor portal (investors.sgx.com)

The investor portal uses a different CMS queryId prefix (9b5f40321735d88857c12bd08459b5aba7fe2d1e) and a different Origin/Referer:

Origin: https://investors.sgx.com
Referer: https://investors.sgx.com/

Some calls (e.g. snapshotreports, announcements/securitycode) reuse the same authorizationtoken header used on www.sgx.com.

Per-stock quote and charts (SECURITIES_API_URL)

Call	URL pattern	Purpose
Stock details	https://api.sgx.com/securities/v1.1/stocks/code/{code}?ts={timestamp}	Per-stock quote/details.
Historic chart	https://api.sgx.com/securities/v1.1//charts/historic/stocks/code/{code}/{1w,1m,1y,5y}?params=trading_time,vl,lt	Historic OHLC-like time-series.
Intraday chart	https://api.sgx.com/securities/v1.1//charts/intraday/stocks/code/{code}/1d?params=trading_time,vl,lt	Intraday time-series.

Fundamentals and quant data (v2.0 endpoints)

These endpoints are not listed in the original appconfig.json but are called by investors.sgx.com:

Endpoint	URL pattern	Purpose
Stock screener	https://api.sgx.com/stockscreener/v2.0/all?params=exchange,exchangeCountryCode,companyName,stockCode,marketCapitalization,salesTTM,priceToEarningsRatio,dividendYield,fourWeekPricePercentChange,thirteenWeekPricePercentChange,twentySixWeekPricePercentChange,fiftyTwoWeekPricePercentChange,netProfitMargin,returnOnAvgCommonEquity,priceToCashFlowPerShareRatio,totalDebtToTotalEquityRatio,salesPercentageChange,sector,priceToBookRatio,priceCurrCode	Cross-sectional quant screener (~701 stocks) with market cap, P/E, yield, returns, sector, etc.
Analyst / valuation	https://api.sgx.com/ratiosreports/v2.0/countryCode/SGP/stockCode/{code}?params=...	Market cap, P/E, EPS, margins, ROA/ROE, debt/equity, dividend yield, beta, 52-week range, analyst consensus, target price.
Snapshot	https://api.sgx.com/snapshotreports/v2.0/countryCode/SGP/stockCode/{code}?params=enterpriseValue,returnOnEquity,companyName,stockCode,sectorName,industryName,reportingCurrency,tradedCurrency,currentFiscalYear,currentFiscalYearEndMonth,currencyIdForMarketCap	Enterprise value, ROE, sector/industry, reporting/traded currency.
Shareholders	https://api.sgx.com/shareholdersreports/v2.0/stockCode/{code}?params=investorName,investorType,investorHoldingsDate,pctOfSharesOutstanding,sharesHeld,sharesHeldChange,turnoverRating	Major shareholder holdings.
Balance sheet	https://api.sgx.com/financialstatementreports/v2.0/balanceSheet/countryCode/SGP/stockCode/{code}?params=all	Balance sheet.
Cash flow	https://api.sgx.com/financialstatementreports/v2.0/cashFlow/countryCode/SGP/stockCode/{code}?params=all	Cash flow statement.
Income statement	https://api.sgx.com/financialstatementreports/v2.0/incomeStatement/countryCode/SGP/stockCode/{code}?params=all	Income statement.

Per-stock corporate actions and announcements

Endpoint	URL pattern	Purpose
Corporate actions	https://api.sgx.com/corporateactions/v1.0?pagesize=10&pagestart=0&ibmcode={ibmcode}&params=id,anncType,dateAnnc,exDate,name,particulars,recDate,datePaid&order=desc&orderBy=dateAnnc	Corporate actions for a given ibmcode.
Announcements	https://api.sgx.com/announcements/v1.1/securitycode?value={code}&cat=ANNC&securityCodeParams=securitycode&sub=ANNC17&pagestart=0&pagesize=250&periodstart={ts}	Per-stock announcements filtered by security code.

CMS document list

https://api2.sgx.com/content-api?queryId=0c6b763562fa9ad6d4b40558d2670db60437f7a6:document_list&variables={...} is used to fetch document metadata such as the IP-CA-TABLE JSON (https://api2.sgx.com/sites/default/files/2026-06/IP-CA-TABLE-20260619_1.json).

Additional CMS operations on www.sgx.com

• data_settlement_prices_list — derivatives settlement prices; variables include underlyingProductsCategory and filter flags.
• prices_list — general prices document list.

Fixed-income / bond endpoints (SECURITIES_API_URL)

Call	URL pattern	Purpose
Bonds snapshot	https://api.sgx.com/securities/v1.1/bonds?params=nc,adjusted-vwap,bond_accrued_interest,...,vwap-currency	All bonds.
Retail bond detail	https://api.sgx.com/securities/v1.1/retailbonds/code/{code}?params=...	Single retail bond.
Retail preference share	https://api.sgx.com/securities/v1.1/retailpreferenceshares/code/{code}?params=...	Single retail preference share.
Bond chart	https://api.sgx.com/securities/v1.1/charts/historic/retailbonds/code/{code}/{period}?params=...	Historic bond chart.

Legacy infofeed endpoints (api3.sgx.com)

The V1_DERIVATIVES_DAILY_LIST_URL and V1_DERIVATIVES_HISTORICAL_LIST_URL endpoints accept an additional noCache={timestamp} parameter:

https://api3.sgx.com/infofeed/Apps?A=COW_Infopubdtstat_Content&B=DailyDataDownload&S_T=1&C_T=60&noCache={timestamp}
https://api3.sgx.com/infofeed/Apps?A=COW_Tickdownload_Content&B=TimeSalesData&C_T=20&noCache={timestamp}

Corporate Actions (CORPORATE_ACTIONS_API_URL = https://api.sgx.com/corporateactions/v1.0)

Call	URL pattern	Purpose
Filter metalist	.../metalist	companyName filter values for the UI dropdown.
Global list	...?pagestart=0&pagesize=20&params=id,anncType,datePaid,exDate,name,particulars,recDate	Paginated global corporate-actions feed.
Per-stock list	...?pagesize=10&pagestart=0&ibmcode={ibmcode}&params=id,anncType,dateAnnc,exDate,name,particulars,recDate,datePaid&order=desc&orderBy=dateAnnc	Corporate actions for a single IBM code.

Circulars (CIRCULARS_API_URL = https://api.sgx.com/circulars/v1.0)

Call	URL pattern	Purpose
Filter metalist	.../metalist	companyName filter values.
List	...?pagestart=0&pagesize=5	Prospectus and supplement/circular records (id, documentDate, prospectusType, subject, companyName).

Meeting Schedules (MEETING_SCHEDULES_API_URL = https://api.sgx.com/meetingschedules/v1.0)

Call	URL pattern	Purpose
Filter metalist	.../metalist	issuerName, securityName, anncType filter lists.
List	...?pagestart=0&pagesize=20&params=issuerName,securityName,meetingDateTime,anncType,anncTitle,recDateTime	Paginated AGM/EGM schedule records.

Corporate Information (CORPORATE_INFORMATION_API_URL = https://api.sgx.com/corporateinformation/v1.0)

Call	URL pattern	Purpose
Filter metalist	.../metalist	issuerName, securityName, industry, country filter lists.
List	...?pagestart=0&pagesize=20&params=issuerName,securityName,industry,country,incorporationDate,listedDate,tradingCurrency,url,email	Paginated issuer profile records.

Wholesale Bonds (WHOLESALE_BONDS_API_URL = https://api.sgx.com/wholesalebonds/v1.0)

Call	URL pattern	Purpose
Filter metalist	.../metalist	issuerName, securityName filter lists.
List	...?pagestart=0&pagesize=20&params=issuerName,securityName,bondType,lastTradedPrice,lastTradedDate,lastTradedYield,couponRate,maturityDate,bondPriceType	Paginated wholesale bond price/yield records.

Evaluated Bond Prices (EVALUATED_BONDS_PRICES_API_URL = https://api.sgx.com/evaluatedbondprices/v1.0)

Call	URL pattern	Purpose
Filter metalist	.../metalist	issuerName, securityName filter lists.
List	...?pagestart=0&pagesize=20&params=issuerName,securityName,cleanPrice,dirtyPrice,evaluatedPriceDate,bidPrice,offerPrice,yield,couponRate,maturityDate	Paginated evaluated bond prices.

Negotiated Large Trades (NLT_API_URL = https://api.sgx.com/negotiatedlargetrades/v1.0)

Call	URL pattern	Purpose
List	...?pagestart=0&pagesize=20	Paginated NLT records (contractCode, instrumentGroup, category, month, year, strikePrice, quantity, businessDate, tradedDate, clearedOnDate, premium, session).

Structured Warrants (STRUCTURED_PRODUCTS_API_URL = https://api.sgx.com/marketmetadata/v2/structure-warrants)

Call	URL pattern	Purpose
Metadata	...?pagestart=0&pagesize=...	Structured-warrant instrument metadata (1,183 records).