OpenClaw/sgx_scrapper
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
sgx_scrapper/adv-investigation
T
History

README.md

Advanced Investigation

This folder holds the detective work we did to figure out how SGX's website talks to its own data APIs.

Why this exists

The SGX website does not publish a clean, public API manual. Instead, it has a list of API endpoints buried in a config file. Our first attempt (Doc/ — previously called discovery/) simply knocked on every endpoint door and wrote down which ones opened and which ones did not. That gave us a rough map, but many doors were locked.

This folder contains the follow-up work: we opened a real web browser, visited the SGX site, and watched exactly which API calls the site makes when you click around. By copying what the website itself does, we found the missing keys: the right URL paths, the right settings, the right headers, and the secret token some APIs demand.

What you will find here

  • investigation.md — the full detective report. It lists, page by page, what we saw in the browser and what it taught us about each API.
  • payloads/ — the raw responses we captured from the website. These are the proof behind the report.
  • README.md — this file.

Simple summary of what we learned

  1. The website starts by loading a config file. Before asking for stock prices or announcements, the SGX site fetches https://www.sgx.com/config/appconfig.json?v=.... That file contains a version number (v) and a CMS version hash. If the version changes, the API calls may change too, so it is worth checking this file first.
  2. Some APIs need extra information. A bare API URL often fails. The site always adds a path, filters, or a params list to say exactly which fields it wants back.
  3. Some APIs need a token. The announcements API, for example, will not talk to you unless you send a special header called authorizationtoken. We traced where the website gets that token: it asks a CMS endpoint called we_chat_qr_validator, receives a scrambled string, and unscrambles it with ROT13.
  4. CORS matters. The APIs check where the request came from. You must send Origin and Referer headers that look like you are calling from https://www.sgx.com or https://investors.sgx.com.

Tools we used

  • Chrome + chrome-devtools-mcp: we browsed the real SGX website and recorded the network traffic.
  • ../scripts/inspect_cache.py: a helper that reads the saved API responses and prints a quick summary of what is inside.
  • Small Python snippets: used on the fly to decode tokens, search JavaScript bundles, and double-check headers.

Relationship to the rest of the project

  • Doc/ — the simplified API field guide. It tells you which endpoints work and how to call them. We are not going to re-run the original discovery sweep; this folder is now the maintained reference.
  • adv-investigation/ — the behind-the-scenes notes on how we figured those calls out.

Working endpoint patterns (from browser recon, 2026-06-25)

Full capture logs and response bodies are in adv-investigation/investigation.md and adv-investigation/payloads/.

Required headers for API replay

Most api.sgx.com, api2.sgx.com, and api3.sgx.com calls require:

Origin: https://www.sgx.com
Referer: https://www.sgx.com/

api.sgx.com/announcements/v1.1/* additionally requires an authorizationtoken header. The token observed in the current frontend bundle is documented in adv-investigation/investigation.md; it should be treated as a credential and not committed in plain text.

Field projection via params

Securities, derivatives, and indices endpoints accept a params query string of comma-separated field abbreviations. Examples of abbreviations seen in browser traffic:

  • Securities: nc, n, type, ls, m, sc, bl, sip, ex, ej, clo, cr, cur, el, r, i, cc, ig, lf, adjusted-vwap, bond_accrued_interest, bond_clean_price, bond_dirty_price, bond_date, b, bv, p, c, change_vs_pc, change_vs_pc_percentage, cx, cn, dp, dpc, du, ed, fn, h, iiv, iopv, lt, l, o, p_, pv, ptd, s, sv, trading_time, v_, v, vl, vwap, vwap-currency.
  • Indices: lp, trading_time, vl.
  • Derivatives: base-date, total-volume, last-traded-price-abs.

Omitting params may return an error or a different response shape.

Securities (SECURITIES_API_URL = https://api.sgx.com/securities/v1.1)

Call URL pattern Purpose
Market aggregate .../aggregate/ Total volume/value and advancer/decliner counts.
Volume by type .../aggregate/volume?securitytype=stocks Total stock volume.
Value by type .../aggregate/value?securitytype=stocks Total stock value.
Sector heat map .../sectorbreakdown 20 SSIC sectors with change/changePercentage.
Circuit breakers .../circuitbreaker Active circuit-breaker records.
Top advancers .../advancers/{stocks,reits,etfs,structuredwarrants,dlcertificates}/{N}/v?params=... Top N by value.
ETF snapshot .../etfs?params=n,nc,cx,lt,c,p,vl,v,sc ETF screener price data.
Full screener ...?excludetypes=bonds&pagestart=0&pagesize=500&sort=vl&sortdir=desc&params=... Paginated, sorted securities table.
Reference snapshot ...?params=nc,n,type,ls,m,sc,bl,sip,ex,ej,clo,cr,cur,el,r,i,cc,ig,lf Static-ish reference data.
Full quote ...?params=nc,adjusted-vwap,bond_accrued_interest,bond_clean_price,bond_dirty_price,bond_date,b,bv,p,c,change_vs_pc,change_vs_pc_percentage,cx,cn,dp,dpc,du,ed,fn,h,iiv,iopv,lt,l,o,p_,pv,ptd,s,sv,trading_time,v_,v,vl,vwap,vwap-currency Rich pricing/volume snapshot.

Derivatives (DERIVATIVES_API_URL = https://api.sgx.com/derivatives/v1.0)

Call URL pattern Purpose
Total volume ...?aggregate=true&aggregate-option=total-volume Single aggregate number.
Futures by commodity ...?groupby=cc&category=futures All futures grouped by commodity code.
History by symbol .../history/symbol/{symbol}?days=10d&category=futures&params=... 10-day OHLC-like history.

Indices (INDICES_API_URL = https://api.sgx.com/indices/v1.0/)

Call URL pattern Purpose
Master list ... All 854 indices with current OHLC/change.
Metadata .../pid/{pid}/ Last price and metadata for a single index.
Intraday chart .../charts/intraday/pid/{pid}/{period}?params=lp,trading_time,vl Intraday time-series.
Historic chart .../charts/historic/pid/{pid}/{period}?params=lp,trading_time,vl Daily historic time-series.

Periods seen: 5d, 1m. PIDs seen: .STI, SGN50N, .SREIT, SGLC40CN, .APACFINN, .EMASIAUN, ASIATECN, IEVNTR, .FTFSTM, .FTFSTS.

Third-party indices (THIRDPARTY_INDICES_API_URL = https://api.sgx.com/thirdpartyindices/v1.0/)

Call URL pattern Purpose
ChinaBond .../chinabonds ChinaBond index data.

Announcements (ANNOUNCEMENTS_API_URL = https://api.sgx.com/announcements/v1.1/)

Call URL pattern Purpose
Issuer filter .../companylist All issuing companies.
Security filter .../securitylist All securities.
Count .../count?periodstart={ts}&periodend={ts} Count in date window.
List .../?periodstart={ts}&periodend={ts}&pagestart=0&pagesize=20 Paginated announcement records.
Summary .../summary/company?periodstart={ts}&cat={cat} Category summary (ANNC, CACT, PLST, TRAD).

IPO (IPO_PERFORMANCE_API_URL and IPO_PROSPECTUS_API_URL)

Call URL pattern Purpose
Years https://api.sgx.com/ipoperformance/v1.0/yearlist Filter dropdown.
Issuers https://api.sgx.com/ipoperformance/v1.0/issuerlist Filter dropdown.
Companies https://api.sgx.com/ipoperformance/v1.0/companylist Filter dropdown.
Performance by year https://api.sgx.com/ipoperformance/v1.0?year=2026 IPO performance data.
Prospectus https://api.sgx.com/ipoprospectus/v1.0/?closing_time={yyyyMMdd_HHmmss} Open/closed IPO issues.

ETF screener (ETF_API_URL = https://api.sgx.com/etfscreener/v1.0)

Call URL pattern Purpose
Documents .../documents ETF factsheet/document metadata.

CMS (CMS_API_URL = https://api2.sgx.com/content-api)

Calls take the form:

https://api2.sgx.com/content-api?queryId={CMS_VERSION}:{operation}&variables={urlencoded_json}

Current CMS_VERSION (from appconfig): 70f75ec90c030bab34d750ee55d74b016f70d4b6.

Operations used by the frontend:

  • page — page-level CMS content; variables include path and lang.
  • all_menus — site navigation; variables: lang.
  • alerts — global alert banners; variables: lang.
  • advertisement_list — ads.
  • derivatives_products_list — derivatives product list; variables: limit, offset, lang.
  • index_list — index taxonomy; variables: lang.
  • market_updates_list — market news list; variables: category, limit, categoryFilterEnabled, lang.
  • taxonomy_terms — vocabulary terms; variables: vid, lang.
  • we_chat_qr_validator — tiny validation call.

Financial reports

FINANCIAL_REPORTS_API_URL should be called paginated:

https://api.sgx.com/financialreports/v1.0?pagestart={n}&pagesize=2000&params=id,companyName,documentDate,securityName,title,url

Investor portal (investors.sgx.com)

The investor portal uses a different CMS queryId prefix (9b5f40321735d88857c12bd08459b5aba7fe2d1e) and a different Origin/Referer:

Origin: https://investors.sgx.com
Referer: https://investors.sgx.com/

Some calls (e.g. snapshotreports, announcements/securitycode) reuse the same authorizationtoken header used on www.sgx.com.

Per-stock quote and charts (SECURITIES_API_URL)

Call URL pattern Purpose
Stock details https://api.sgx.com/securities/v1.1/stocks/code/{code}?ts={timestamp} Per-stock quote/details.
Historic chart https://api.sgx.com/securities/v1.1//charts/historic/stocks/code/{code}/{1w,1m,1y,5y}?params=trading_time,vl,lt Historic OHLC-like time-series.
Intraday chart https://api.sgx.com/securities/v1.1//charts/intraday/stocks/code/{code}/1d?params=trading_time,vl,lt Intraday time-series.

Fundamentals and quant data (v2.0 endpoints)

These endpoints are not listed in the original appconfig.json but are called by investors.sgx.com:

Endpoint URL pattern Purpose
Stock screener https://api.sgx.com/stockscreener/v2.0/all?params=exchange,exchangeCountryCode,companyName,stockCode,marketCapitalization,salesTTM,priceToEarningsRatio,dividendYield,fourWeekPricePercentChange,thirteenWeekPricePercentChange,twentySixWeekPricePercentChange,fiftyTwoWeekPricePercentChange,netProfitMargin,returnOnAvgCommonEquity,priceToCashFlowPerShareRatio,totalDebtToTotalEquityRatio,salesPercentageChange,sector,priceToBookRatio,priceCurrCode Cross-sectional quant screener (~701 stocks) with market cap, P/E, yield, returns, sector, etc.
Analyst / valuation https://api.sgx.com/ratiosreports/v2.0/countryCode/SGP/stockCode/{code}?params=... Market cap, P/E, EPS, margins, ROA/ROE, debt/equity, dividend yield, beta, 52-week range, analyst consensus, target price.
Snapshot https://api.sgx.com/snapshotreports/v2.0/countryCode/SGP/stockCode/{code}?params=enterpriseValue,returnOnEquity,companyName,stockCode,sectorName,industryName,reportingCurrency,tradedCurrency,currentFiscalYear,currentFiscalYearEndMonth,currencyIdForMarketCap Enterprise value, ROE, sector/industry, reporting/traded currency.
Shareholders https://api.sgx.com/shareholdersreports/v2.0/stockCode/{code}?params=investorName,investorType,investorHoldingsDate,pctOfSharesOutstanding,sharesHeld,sharesHeldChange,turnoverRating Major shareholder holdings.
Balance sheet https://api.sgx.com/financialstatementreports/v2.0/balanceSheet/countryCode/SGP/stockCode/{code}?params=all Balance sheet.
Cash flow https://api.sgx.com/financialstatementreports/v2.0/cashFlow/countryCode/SGP/stockCode/{code}?params=all Cash flow statement.
Income statement https://api.sgx.com/financialstatementreports/v2.0/incomeStatement/countryCode/SGP/stockCode/{code}?params=all Income statement.

Per-stock corporate actions and announcements

Endpoint URL pattern Purpose
Corporate actions https://api.sgx.com/corporateactions/v1.0?pagesize=10&pagestart=0&ibmcode={ibmcode}&params=id,anncType,dateAnnc,exDate,name,particulars,recDate,datePaid&order=desc&orderBy=dateAnnc Corporate actions for a given ibmcode.
Announcements https://api.sgx.com/announcements/v1.1/securitycode?value={code}&cat=ANNC&securityCodeParams=securitycode&sub=ANNC17&pagestart=0&pagesize=250&periodstart={ts} Per-stock announcements filtered by security code.

CMS document list

https://api2.sgx.com/content-api?queryId=0c6b763562fa9ad6d4b40558d2670db60437f7a6:document_list&variables={...} is used to fetch document metadata such as the IP-CA-TABLE JSON (https://api2.sgx.com/sites/default/files/2026-06/IP-CA-TABLE-20260619_1.json).

Additional CMS operations on www.sgx.com

  • data_settlement_prices_list — derivatives settlement prices; variables include underlyingProductsCategory and filter flags.
  • prices_list — general prices document list.

Fixed-income / bond endpoints (SECURITIES_API_URL)

Call URL pattern Purpose
Bonds snapshot https://api.sgx.com/securities/v1.1/bonds?params=nc,adjusted-vwap,bond_accrued_interest,...,vwap-currency All bonds.
Retail bond detail https://api.sgx.com/securities/v1.1/retailbonds/code/{code}?params=... Single retail bond.
Retail preference share https://api.sgx.com/securities/v1.1/retailpreferenceshares/code/{code}?params=... Single retail preference share.
Bond chart https://api.sgx.com/securities/v1.1/charts/historic/retailbonds/code/{code}/{period}?params=... Historic bond chart.

Legacy infofeed endpoints (api3.sgx.com)

The V1_DERIVATIVES_DAILY_LIST_URL and V1_DERIVATIVES_HISTORICAL_LIST_URL endpoints accept an additional noCache={timestamp} parameter:

https://api3.sgx.com/infofeed/Apps?A=COW_Infopubdtstat_Content&B=DailyDataDownload&S_T=1&C_T=60&noCache={timestamp}
https://api3.sgx.com/infofeed/Apps?A=COW_Tickdownload_Content&B=TimeSalesData&C_T=20&noCache={timestamp}

Corporate Actions (CORPORATE_ACTIONS_API_URL = https://api.sgx.com/corporateactions/v1.0)

Call URL pattern Purpose
Filter metalist .../metalist companyName filter values for the UI dropdown.
Global list ...?pagestart=0&pagesize=20&params=id,anncType,datePaid,exDate,name,particulars,recDate Paginated global corporate-actions feed.
Per-stock list ...?pagesize=10&pagestart=0&ibmcode={ibmcode}&params=id,anncType,dateAnnc,exDate,name,particulars,recDate,datePaid&order=desc&orderBy=dateAnnc Corporate actions for a single IBM code.

Circulars (CIRCULARS_API_URL = https://api.sgx.com/circulars/v1.0)

Call URL pattern Purpose
Filter metalist .../metalist companyName filter values.
List ...?pagestart=0&pagesize=5 Prospectus and supplement/circular records (id, documentDate, prospectusType, subject, companyName).

Meeting Schedules (MEETING_SCHEDULES_API_URL = https://api.sgx.com/meetingschedules/v1.0)

Call URL pattern Purpose
Filter metalist .../metalist issuerName, securityName, anncType filter lists.
List ...?pagestart=0&pagesize=20&params=issuerName,securityName,meetingDateTime,anncType,anncTitle,recDateTime Paginated AGM/EGM schedule records.

Corporate Information (CORPORATE_INFORMATION_API_URL = https://api.sgx.com/corporateinformation/v1.0)

Call URL pattern Purpose
Filter metalist .../metalist issuerName, securityName, industry, country filter lists.
List ...?pagestart=0&pagesize=20&params=issuerName,securityName,industry,country,incorporationDate,listedDate,tradingCurrency,url,email Paginated issuer profile records.

Wholesale Bonds (WHOLESALE_BONDS_API_URL = https://api.sgx.com/wholesalebonds/v1.0)

Call URL pattern Purpose
Filter metalist .../metalist issuerName, securityName filter lists.
List ...?pagestart=0&pagesize=20&params=issuerName,securityName,bondType,lastTradedPrice,lastTradedDate,lastTradedYield,couponRate,maturityDate,bondPriceType Paginated wholesale bond price/yield records.

Evaluated Bond Prices (EVALUATED_BONDS_PRICES_API_URL = https://api.sgx.com/evaluatedbondprices/v1.0)

Call URL pattern Purpose
Filter metalist .../metalist issuerName, securityName filter lists.
List ...?pagestart=0&pagesize=20&params=issuerName,securityName,cleanPrice,dirtyPrice,evaluatedPriceDate,bidPrice,offerPrice,yield,couponRate,maturityDate Paginated evaluated bond prices.

Negotiated Large Trades (NLT_API_URL = https://api.sgx.com/negotiatedlargetrades/v1.0)

Call URL pattern Purpose
List ...?pagestart=0&pagesize=20 Paginated NLT records (contractCode, instrumentGroup, category, month, year, strikePrice, quantity, businessDate, tradedDate, clearedOnDate, premium, session).

Structured Warrants (STRUCTURED_PRODUCTS_API_URL = https://api.sgx.com/marketmetadata/v2/structure-warrants)

Call URL pattern Purpose
Metadata ...?pagestart=0&pagesize=... Structured-warrant instrument metadata (1,183 records).
Reference in New Issue View Git Blame Copy Permalink