Life is a File 📁

Daily Market Analysis - March 2021

2021-03-01T00:00:00+00:00

01 Mar, 2021 Monday

Nifty50

OHLC - 
% change - 
Candle - 
VIX - 
daily RSI - 
Support levels - 
Resistance levels - 
Daily candlestick chart - 
Weekly candlestick chart - 
FII v/s DII -

Daily Market Analysis - February 2021

2021-02-01T00:00:00+00:00

01 Feb, 2021 Monday | Budget Day

The market witnessed selling pressure for six consecutive sessions, with the benchmark indices losing over a percent each in the week gone by. The BSE Sensex slipped 2,592.77 points, or 5.3 percent, to end at 46,285.77 and the Nifty50 fell 737.3 points or 5 percent to finish at 13,634.60 In the last week, the 30-share BSE benchmark tumbled 2,592.77 points or 5.30 per ent due to profit-booking ahead of the Union Budget. The short term trend of Nifty continues to be weak. The negation of the upside bounce attempt and the downside break of weekly support signals more declines for the Nifty. The Union Budget could play a vital role to show the direction to the market. Any rise from here can face resistance at around 13,750-13,800. The near- term downside targets to be watched for is around 13,050 (20-week EMA). The weak closing of Nifty-50 below 14,000-mark on the expiry day suggests more weakness in the market if the budget disappoints. We need to wait and see how the Nifty closes on February 1 after the Budget to get the next directional move. On the upside, the recent peak of around 14,645 could be the major resistance. The economic survey failed to trigger the rebound in the markets and now all eyes are on the Union Budget. The markets were successful in breaking 13,700 and closed below it too. It can slide further to 13,400 and 13,200. Any rally up can be utilised to short the Nifty for lower targets. The resistance is now at 14,000 and until that is not crossed, we will remain in the grip of the bears.

It was a highest Budget day gain reported by the Sensex in last 22 years as it rose 5 percent, while Nifty rose 4.7 percent. Nifty forms Long White Day candle on Budget day; experts say index can march towards lifetime high. The Sensex rallied 2,300 points, or 5 percent, to 48,600.61 and Nifty gained 646.60 points, or 4.74 percent, at 14,281.20 in today’s session. This has been the best performance by the stock market on a Budget Day since 1999. Gains were mainly led by banking, infra, metal and auto stocks. Bank Nifty touched a record high with a gain of 8 percent closing at 33,305.30 on FM measures to clean up the NPAs in the sector. The Nifty50 ended 646.6 points or 4.74 percent higher at 14,281.20 and the BSE Sensex climbed 2,315 points or 5 percent higher at 48,600.61. The rally was led by banking stock as Nifty Bank surged 8.26 percent, after touching record high levels, to end at 33.089.05. Markets heaved a sigh of relief in absence of any major change in personal or corporate taxes and therefore reacted positively. The market is likely to remain highly volatile amid Union Budget and the ongoing earning season The Nifty50 rebounded sharply with a 5 percent gains on February 1 snapping the six-day losing streak as investors cheered Budget proposals made by Finance Minister Nirmala Sitharaman. The stellar rally was driven by banking & financials, auto and metals stocks. The index closed decisively above 14,200 levels and formed a large bullish candle which resembles Long White Day pattern on the daily charts. bullish momentum to continue in the near term but small profit booking can’t be ruled out after a sharp rally. The Nifty50 opened strong at 13,758.60 and extended gains as the day progressed to hit a high of 14,336.35. The index closed 646.60 points or 4.74 percent higher at 14,281.20. In next trading sessions, if bulls manage to push the index beyond 14,336 levels, upswing may expand to 14,521. Bank Nifty opened gap up at 30,976.35 and saw the index touching its life-time high levels of 33,305. The banking index registered its highest intraday gains by sky rising and closing with gains of 2,523.60 points or 8.26 percent at 33,089.10. All the banking stocks peaked and bullishness continued throughout the day. The index formed a strong bullish candle on the daily scale and continued its formation of higher top - higher bottom seen since the last two sessions. Bank Nifty has to hold 32,500 to witness an up-move towards 33,333 and 34,000 levels, while on the downside, support is seen at 32,000 and 31,750 levels Except pharma, all other sectoral indices ended with gains of 1-8 percent each. BSE Midcap and Smallcap rose 2-3 percent each. Banking stocks rally - Banks rallied the most on February 1 after Finance Minister Nirmala Sitharaman said the government would infuse Rs 20,000 crore into public sector banks (PSBs) in 2021-22, to meet the regulatory norms. The Finance Minister announced the setting up of an entity to address the stressed assets of banks. An organization will be set up through an Asset Reconstruction Company (ARC) model, she said in the 2021 Budget Speech. She also proposed to divest two PSU banks and one general insurance company in FY22. IndusInd Bank share price jumped over 15 percent followed by ICICI Bank which added 13 percent and State Bank of India which was up 11 percent. The other gainers included Bank of Baroda, Axis Bank and Punjab National Bank.

02 Feb, 2021 Tuesday

Indian stock market continues its bull run after investors gave a thumbs up to Budget 2021. Bank Nifty jumped 4 percent after surging over 8 percent on Budget day. The index hit new record high of 34,652.50 at 09:42 hours. Bank Nifty has been bucking the trend since last couple of sessions, while the broader market was correcting brutally. The markets ended up by a strong 2.5%. This comes after yesterday’s massive 5% gain. The market extended Budget-led gains for the second consecutive session on February 2 with the Nifty50 moving closer to record high levels, backed by buying in all sectors but FMCG. Foreign institutional investors (FIIs) net bought shares worth Rs 6,181.56 crore, whereas domestic institutional investors (DIIs) net sold shares worth Rs 2,035.2 crore in the Indian equity market on February 2 Sensex jumps nearly 1,200 points, Nifty reclaims 14,700; 5 factors behind market rally

03 Feb, 2021 Wednesday

The formation of long bull candles and a possibility of faster retracement of downswing is expected to pull the Nifty beyond 14,750-14,800 levels in the next 1-2 sessions The broader markets also participated in the bull run. The Nifty Midcap 100 index gained 2.4 percent and Smallcap 100 index was up 1.23 percent. According to pivot charts, the key support levels for the Nifty are placed at 14,500.77, followed by 14,353.73. If the index moves up, the key resistance levels to watch out for are 14,763.27 and 14,878.73. In the next trading session, unless Nifty gets past 14,731 levels strength should not be expected. In case if the said hurdle is cleared then Nifty may register a new lifetime high which could extend to 15200 levels Considering the strong up move of last two sessions, traders are advised to avoid fresh buying in the index and preference should be given to booking profits on long positions Trends on SGX Nifty indicate a positive opening for the index in India with a 50 points gain. The S&P BSE Sensex rallied 1,197.11 points or 2.46 percent to 49,797.72 on February 2 while the Nifty50 climbed 366.70 points or 2.57 percent to 14,647.90 levels. Foreign institutional investors (FIIs) net bought shares worth Rs 6,181.56 crore, whereas domestic institutional investors (DIIs) net sold shares worth Rs 2,035.2 crore in the Indian equity market on February 2 Not only largecaps, but mid & smallcap stocks also participated in the rally. NiftyBank touched a fresh record high above 34000 for the first time. Experts advise investors to stay with stocks in economy-lined sectors that are likely to gain the most from Budget proposals. Nifty Bank skyrockets 600 pts on broad-based buying; Nifty PSU Bank in fast lane; IndusInd Bank spurts 9% Nifty PSU Bank soars 2.33% at 2,071.15. While Nifty Private Bank zooms 1.88% and was trading at 18,937. Led by positive markets rally, corporate earnings and Union Budget 2021, the overall banking sector witnessed huge buying sentiment in Wednesday’s trading session. PSU Banks were in the fast lane, while private banks added substantial gains.

04 Feb, 2021 Thursday

Markets pullback from record-highs amid weak global cues; Sensex shed 240 pts, Nifty struggles at 14,730; Banks underperform The benchmarks were dragged due to huge selloff in banking stocks followed by consumer durables stocks. The Indian markets entered into negative territory during Thursday’s trading session, with Sensex and Nifty 50 pulling back from the previous record-high gains. The benchmarks were dragged due to huge selloff in banking stocks followed by consumer durables stocks. IT and financial stocks also took a toll, however, at a slower pace. Investors traded cautiously amid weaker global cues. Nifty Bank dives amid RBI policy meet; Nifty PSU Bank rally; Heavyweights IndusInd, ICICI Bank, HDFC Bank drag RBI governor Shaktikanta Das along with members of monetary policy committees (MPC) have begun policy meet from February 03, and the final outcome will be announced on February 05, 2021. After Budget 2021, investors are closing gauging RBI’s monetary policy announcement which is set to be presented tomorrow. After three days of catching up with the new expectations raised by the Budget, the stock market seems to be settling back into a normal rhythm. The heavy lifting that the RBI wanted the government to do, in terms of fiscal support to growth, has been done at least in intention. Tomorrow, the RBI’s credit policy statement, stance and rate decision will reveal what it makes of where we stand.

05 Feb, 2021 Friday

A historical week for Indian markets! Both Sensex and Nifty climbed crucial psychological peaks surging over 9 percent in the week that ended on February 5. The S&P BSE Sensex climbed Mount 51K in intraday trade while the Nifty50 climbed above 15000 levels for the first time. The S&P BSE Sensex hit a peak of 51,073, while the Nifty50 surged to a high of 15,014 in the week gone by. However, the broader markets underperformed. Growth-oriented and market-friendly Union Budget 2021 kept bulls happy during the week. The Sensex and Nifty ended in the green for five consecutive sessions, rising almost 10 percent for the week. Now, the Nifty50 has to continue to hold above 14,864 zones to continue its bullish momentum towards 15,250-15,500 levels. Support can be seen around 14400 levels Indian benchmark indices rose more than 9 percent in the week ended February 5 on the back positive global cues and supportive domestic events. The rally was kickstarted by the growth oriented Union Budget on February 1, while better quarterly earnings from India Inc, continued FIIs buying and RBI Monetary Policy kept the party going on the D-Street through the week. During the week, benchmark indices touched their fresh record highs, with Sensex crossed the 51,000 mark and Nifty50 usurping 15,000. A buy on dips strategy would be advisable. There is ample scope for the markets to correct during intraday sessions. These dips can be utilised to make fresh long positions for higher targets. This way the risk reward trade off would be favourable BankNifty rallied by over 15 percent this week largely driven by Monday’s Big Fat Green Bar as it eclipsed multiple tall red bars seen since January 21, 2020. we are likely to enter a phase of outperformance of banks.

08 Feb, 2021 Monday

The just-concluded RBI policy meet reaffirms our conviction that growth is coming back. Along the way, challenges remain, even as the Budget has raised expectations significantly. In the coming week, a batch of macro data points is in the pipeline. And not to forget, some big names are flashing on the earnings dashboard The Budget speech is long over, but the feel-good factor is still resonating. The market clocked the biggest ever weekly gains in absolute terms, with the benchmark indices hitting new highs as the bulls, backed by the banking sector, dominated the week ended February 5. Use dips to buy! Nifty50 could test 15,500 in February series The markets achieved a historic feat by attaining the highest-ever jump on Budget Day in a decade. Both, the Nifty50 and Sensex rallies were primarily fuelled by the Budget which brought with it immense optimism and positive economic reforms that aimed at economic growth through infrastructure development, capex incentives, privatisation, and asset monetisation. The market is likely to remain bullish as there is no significant negative news event in the near term. On the downside, 14600 is established as immediate support and any break of the same will lead to short-term weakness in the market. Nifty formed an indecisive ‘Doji’ kind of candle formation on the daily scale and a long bullish candle on the weekly chart on Friday. Continuing the selling spree for the eighth consecutive month, mutual funds pulled out Rs 12,980 crore from equities in January as surge in markets provided an opportunity to book profits. The markets, despite withdrawals from mutual funds in the past few months, have continued to rise as flows from FPIs have been robust. Foreign portfolio investors (FPIs) have put in Rs 19,472 crore in the Indian equity markets in January after investing Rs 1.7 lakh crore in 2020. The surge in the markets has provided investors an opportunity to book profit. That could have led investors redeem their investments, resulting in mutual funds pulling out investments from equity markets in January.

The Nifty opened on a positive note on February 8 and extended higher for the sixth straight session. As a result, it has crossed the 15,000-mark on a closing basis. Going ahead, 15,250 is the key level to watch out for. Nifty has to continue to hold above 15,000 zones to continue its bullish momentum towards 15,250 then 15,500, while support can be seen around 15,000 and then 14,750 zones. The Indian market rallied for a sixth day in a row, pushing benchmark indices to record highs yet again on February 8, as the S&P BSE Sensex hit 51,523 and the Nifty50 15,159. The Sensex rose 617 points to 51,348, while the Nifty50 closed with gains of 191 points at 15,115. Investors are advised to stay long on the index as it is still a buy on dips market. India VIX moved up by 2.30 percent from 23.41 to 23.95 levels. Now, VIX needs to cool down and hold below 21 for the momentum to continue. The Nifty formed a small-bodied bullish candle on a daily scale and continues its higher highs-higher lows formation of the last six sessions. The bulls continued to dominate the benchmark indices following supportive global cues. The Nifty index ended higher by 1.3 percent to close above the 15,100. We are eyeing 15,200 in the Nifty and going forward earnings and global cues will dictate the market trend. Domestic market participants would actively track key macro data like IIP, CPI and WPI data. Overall we expect the market to continue its upward journey on the back of healthy corporate earnings, strong liquidity; positive developments on the vaccine front, broad based economic recovery and low interest rates.

09 Feb, 2021 Tuesday

Sensex, Nifty rally 10% since Budget Day to hit record highs Both Sensex and Nifty50 have rallied by more than 10 percent each since Budget to touch a lifetime high of 51,514, and 15,153, respectively, on February 8. Benchmark indices more than doubled from the March 2020 low caused by the outbreak of the pandemic, but the recovery was swift and the market did not give much time to investors to buy the dip. For investors who are already sitting on profit should use the rally to book some profits, suggest experts. The rally could extend towards 15,500 in February series, and to 16,000 in the year 2021 on the Nifty50 Although the market is high, we don’t see the market is overvalued in terms of the fundamentals but we believe that the market sentiment is at the peak of optimism and an adjustment of it is more likely We recommend the investors liquidate 30-40 percent of their holding at the current level at its maximum and wait for a 6-10 percent correction to re-enter or add into the existing portfolio The Budget 2021 has made most of the sectors attractive. The action will be focused on the economy-related sectors as well as consumption. Investors will not look for stocks that are on a growth path; hence, small & midcaps could do well. Dips should be used to accumulate quality stocks. We have often seen that correction of 5-8% in benchmark indices in a short period of time along with 10-12% decline in quality companies is always a good opportunity to add/accumulate. Invest for the long haul, do not be too greedy or too fearful. If the growth starts to pick up, then mid and smallcaps may start to do well compared to the largecaps.

Indian market snapped a 6-day winning streak and closed flat after hitting a record high on February 9. The S&P BSE Sensex touched a record high of 51,835 while the Nifty50 touched a high of 15,257 during the day. Nifty remained volatile as it traded in a wider range of 150 points but ended on a flattish note. It formed a small Bodied Candle on the daily scale but continues its higher highs - higher lows formation of the last seven sessions market is overstretched and last-hour selling pressure indicated a strong possibility of quick intraday correction from the current levels. Technically, the index has formed a Hammer kind of candlestick pattern which clearly indicates indecisiveness between bulls and bears.

10 Feb, 2021 Wednesday

The market ended flat for the second day in a row in a highly volatile session on February 10. At close, the Sensex was down 19.69 points or 0.04% at 51,309.39, and the Nifty was down 2.80 points or 0.02% at 15,106.50. Previous few weeks have been quite volatile where in Nifty witnessed a 7.5 percent correction and an equally strong bounce back to scale new highs. The medium-term undertone continues to be extremely positive; in the near term, some consolidation is expected before the up move resumes. We believe 14,700-14,800 is a strong entry range for the index and on the higher side one can expect 15,700-16,000. IT and select BFSI stocks look positive while metals and realty are expected to witness consolidation Among sectors, buying witnessed in auto, IT, pharma names, while some selling seen in the banking and infra stocks. Nifty formed a Doji candle on daily scale with a long lower shadow indicating dips were being bought in the market. Nifty has to hold above 15000 to continue its bullish momentum towards 15200 and 15250 zones while on the downside major support can be seen around 14850 and 14750 zones For the next few trading sessions, 15270/51900 should act as a trend decider level, above which we can expect a breakout rally towards 15315-15350/52500-53000 Indian market closed in the red for the second consecutive day in a row on February 10 pushing benchmark indices lower, but a smart bounce back towards the close of the day capped the losses. This suggests that bulls have not left the party yet. The market manages to hold above the Nifty 50 Index support level of 15000. The technical factors are aligned to support a range-bound market movement in the coming week. The Nifty opened on a positive note however faced selling pressure as the day progressed. The index came down to fill up the gap area of 15041-15014. The hourly chart shows that the index moved lower to test the hourly lower Bollinger Band where the selling was absorbed. As long as Nifty shows similar consolidation for the next 1-2 sessions then odds of sharp upside bounce can’t be ruled out Another high wave type candle was formed with minor upper and long lower shadow. Technically, this pattern again signals confusion among participants at the highs, which is now leading to volatility. But the formation of long lower shadow could signal an emergence of sharp buying on dips

11 Feb, 2021 Thursday

12 Feb, 2021 Friday

The market inched closer to record high levels but erased all its gains in the last couple of sessions amid volatility to end flat. Nifty formed a Doji pattern on the daily charts while forming a small bullish candle on the weekly scale. Selling pressure was in FMCG, metals and pharma stocks. However, banking & financials provided strong support.

During the week we saw maximum activity in medium-sized and small-sized companies. Going ahead, these mid and small-size companies could hurt the market if the Nifty50 index fails to break the 15,260 levels. The broader markets outperformed frontliners on the weekly basis. The Nifty Midcap 100 and Nifty Smallcap 100 indices gained 2.18 percent and 3.85 percent

Budget & MPC – Twin Impact on Markets - Post a momentous Budget and MPC week, markets opened with the residual firepower but slowed down and consolidated for most of the week. Despite this, Nifty attained highs of 15,257 for the first time in history.

Nifty50

OHLC - 15186|15243|15081|15163 
% change - −10.0 points (-0.066%)
Candle - DOJI, indecisive candle
VIX - 
daily RSI - 68
Support levels - 15081, followed by 15000
Resistance levels - 15244 and 15325
Key events - 
Analysis for tomorrow - 
Sectoral contributors - 
Daily candlestick chart - indecisiveness in the daily chart
Weekly candlestick chart - small bullish candle
FII v/s DII - FIIs net sold shares worth Rs 37.33 crore, DIIs net sold shares worth Rs 597.62 crore
Comments on futures - high delivery percentage seen in ICICIGI (68%) with a future price of 1497

Nifty Bank

The Nifty Bank smartly outpaced Nifty50, rising 356.80 points or 1 percent to 36,108.90 on February 12. The important pivot level, which will act as crucial support for the index, is placed at 35,765.13, followed by 35,421.36. On the upside, key resistance levels are placed at 36,387.73 and 36,666.57.

On the weekly basis, the Sensex added 812.67 points, or 1.6 percent, to end at 51,544.30, while the Nifty rose 239.05 points, or 1.6 percent, to finish at 15,163.30 levels. The index formed a Doji pattern on the daily charts while forming a small bullish candle on the weekly scale. Bank Nifty formed a bullish candle on the daily scale while there was Doji formation on the weekly scale.

15 Feb, 2021 Monday

For the week ended February 12, Sensex and Nifty50 closed with gains of 1.6 percent. If the Nifty50 index crosses 15,270 for the 15,500 levels, it would be advisable to buy. Fall below 15,100 will lead to further weakness. A break below 15,000 would be an interesting one to participate in. In this scenario, a healthy profit-booking towards 14,600 – 14,400 cannot be ruled out The market is now constrained within the immediate support and resistance of 14970 and 15250 and a break on either side will dictate the trend for the upcoming week.

Based on the chart of largecap companies, the weakness is increasing in the short-term, and it would probably lead to a quick correction to the level of 14,850 or 14,750. According to pivot charts, the key support levels for the Nifty are placed at 15,081.7, followed by 15,000.1. If the index moves up, the key resistance levels to watch out for are 15,244.2 and 15,325.1.

Nifty50 had a gap-up opening today and touched a record high of 15313, after showing signs of consolidation last week. Sensex also reached record high level of 52000 for the first time. stay long but consolidation could continue in the coming week, but at the same time small & midcaps might continue to outperform. The level of 15,300 – 15,000 will be seen as a crucial range and a breakout on either side is likely to dictate the direction of the market. The pragmatic strategy would be to wait for some sort of trend deciding action and then one should be looking to place aggressive bets

Indian share markets benchmarks BSE Sensex and Nifty 50 scaled fresh lifetime highs on Monday, on the back of firm global cues, upbeat Q3 earnings and strong foreign fund inflows Strong global cues and consistent buying by the FIIs helped the Indian market to hit fresh record highs. NBFC sector was in uptrend today. Nifty has strong resistance at 15,300 in the form of maximum concentration of Call OI for this week’s expiry as of now. On technical charts, the zone of 15300-15500 seems to be a very stiff and strong resistance area. One may look for profit booking in financials, rate sensitive and high beta stocks in favor of traditionally defensive sectors going ahead from here. While on the lower side, supports are placed at 15000 and 14850 for the immediate near term. Given the strength of the global markets and the CPI inflation figures, Nifty/Sensex can rise to the level of 15500/52500 levels and that could be the time to shift the portfolio from weak stocks to strong large-cap companies

Nifty50

OHLC - 15270 | 15338 | 15254 | 15335
% change - +172 points (+1.14%)
Candle - strong bullish candle
VIX - 
daily RSI - Above 70
Support levels - 
Resistance levels - 
Key events - 
Analysis for tomorrow - 
Sectoral contributors - Banking sector surged 3.5%
Daily candlestick chart - bullish
Weekly candlestick chart - bullish
FII v/s DII - 
Comments on futures -

16 Feb, 2021 Tuesday

Nifty50 hit a life high of 15431 before investors preferred to book profits at higher levels. The index opened the day with gap-up gains, but was not able to hold the gains for long and witnessed a profit booking and closed the day on a flattish note at 15318. Bears tried to take control of D-Street in the second half of the trading sessions on Tuesday but bulls gave a tough fight and narrowed losses towards the close of the trade. Market, though volatile, kept its momentum during the morning hours, however, failing to hold on to it due to a weak opening of European markets. The downfall was aggravated by private banks, IT, and FMCG stocks while mid and small caps continued their outperformance.

The Nifty 50 Index is still holding above the support level of 15250. It is critical to sustaining above the 15250 Nifty50 Index level to keep the short-term positive trend intact. If nifty50 breaks out above 15370, initiate buying and look for an exit around 15520. Overall, the investor should maintain a stop below 15250. 15250 will act as immediate support followed by 15100 odd levels. If the index holds the above-said levels then we may see the current momentum extend further towards the immediate hurdle zone of 15400-15500 zone.

The recent consolidation in the index is in line with expectations and the market is expected to continue on bullish momentum based on current price action. The weekly price action resulted in a High wave candle indicating a healthy breather after post budget rally measuring more than 23 % rally in just seven sessions the daily stochastic had approached to overbought trajectory which has resulted in a sideways consolidation in the last four sessions.

After a 1,300 point surge on Monday, the Nifty Bank index saw some consolidation to end 200 points or 0.6% lower at 37,098.

Nifty50

OHLC - 15371 | 15431 | 15243 | 15317
% change - +3 points (+0.02%)
Candle - bearish candle
VIX - 
daily RSI - Above 70
Support levels - 15250, 15100
Resistance levels - 15520
Key events - 
Analysis for tomorrow - 
Sectoral contributors - 
Daily candlestick chart - bearish
Weekly candlestick chart - bullish
FII v/s DII - 
Comments on futures -

18 Feb, 2021 Thursday (weekly expiry)

The Indian market consolidated for a third day in a row on February 18, pushing benchmark indices below crucial support levels. The Sensex fell 379 points to 51,324, while the Nifty50 closed 90 points lower at 15,118. The Nifty is trading around 15,110 after trading in the 15,250-15,084 range. Traders are advised to refrain from building a new buying position until there is further improvement and a breakout of 15,370. The Nifty index opened flat and has been facing weakness for the last three trading sessions. The index failed to surpass 15,250 and drifted towards 15,080 levels. It remained negative to range-bound for the most part of the session and closed the day with the losses of around 90 points.

The bearish rally in the market was led by private banks and auto stocks, while PSU banks continued their outperformance on hopes of privatisation Sectorally, action was seen in oil & gas, utilities, public sector, power, and metal, while auto, banks, finance, healthcare, realty and telecom witnessed profit booking. After a sharp run since March 2020 low, there is discomfort on valuations, as these levels are the highest India has traded in the past

My Analysis - Nifty formed a bearish candle for the third consecutive day. Nifty has lost around 300 points in last 3 days from the record highs of 15400. Nifty bank has also lost over 550 points from the record high of 37500 in last 3 sessions. Major support and resistance level for nifty50 are 15000 and 15400 respectively. Market might consolidate and move in the 400-500 points zone also and breakout next week over positive cues. If nifty breaches 15000, major correction can be witnessed. Markets are richly valued and, therefore, are vulnerable to corrections. One can expect Nifty to show upside bounce from the lows of around 15,000 levels in the next 1-2 sessions. On the flip side, a decisive move below 14,950 is likely to trigger more weakness in the near term.

Nifty50

OHLC - 15238 | 15249 | 15078 | 15120
% change - -88 points (-0.58%)
Candle - bearish candle
VIX - 
daily RSI - 62
Support levels - 15085
Resistance levels - 15370
Daily candlestick chart - bearish
Weekly candlestick chart - bearish
FII v/s DII -

19 Feb, 2021 Friday

Stock Market performance this week

Markets kicked-off the week on a positive note, however in the later half of the week it witnessed selling pressure after touching new highs. Sensex, Nifty post weekly loss for 2nd straight week Benchmark indices corrected by around 1.2% this week. Nifty closed 0.9% lower at 14982 today. NIFTY is down 1.2% in the last one week. The Sensex tumbled 435 points. Nifty-50 dipped 137 points, or 0.91 per cent to 14,981.75. The Nifty-50 and BSE Sensex declined 1.2 per cent and 1.27 per cent respectively in the entire week.
Defying the benchmark indices, broader markets have continued to show strength and inched higher in the last few trading session. On the broader front, The BSE Midcap index closed lower by 1.67 per cent, Smallcap index down 0.76 per cent. All sectoral indices closed red as selling was witnessed in all sectors throughout the day. The sentiment turned negative in the last four sessions of the week, as indices closed lower for four consecutive sessions and also erased all the previous gains in the first two trading sessions of the week. BSE Sensex and Nifty 50 ended at nine-day lows on Friday as bears took over the Dalal Street. The Nifty Bank hit its lowest level in 10 trading sessions and closed lower by 2.04%. The biggest contributors to the fall in the banking index were ICICI Bank, HDFC Bank and State Bank of India.

Nifty down 0.7% at 15,012 
Sensex down 0.6% at 51,027 
Nifty Bank down 1.5% at 36,033 
Nifty Mid cap index down 1.5% 
Nifty Small cap index down 0.6% 
India Volatility Index up 2.1% to 22 
PSU Bank index is the top sectoral laggard; down 4.4% 
Nifty Auto index is the other major laggard; down 2.3% 
Nifty Metal and Nifty Pharma index down 1.5% each
UPL remains the top Nifty gainer; up 2.2%
Tata Motors is the top Nifty laggard; down 4.3% 
40 out of the 50 index stocks decline 
1,127 stocks on the NSE decline while 725 trade with gains

Reason behind the market correction

Markets today observed the worst day of the week as traders continue to book profits and weak global cues played the spoilsport. An abrupt and sharp rise in domestic as well as global bond yields led to this selling pressure in the bullish market. Bad performance in the Banking and Financial sectors dragged the market the most in the previous sessions as shares of HDFC Bank, SBI and Bajaj Finance witnessed heavy selling. Rising crude oil prices and fears of rising inflation also impacted the performance of the markets this week. Bond yields are largely inversely proportional to the returns in equity markets. Therefore, when bond yields decline, equity markets tend to outperform and as bond yields rise equity markets returns tend to falter. Hence, such a sharp rise in bond yields this week could be one the major reasons for a break in the equity party.

Where could market possibly head from here next week

Bounce back in Nifty from the lows of 15000 is highly possible, however be cautious if the index falls below 14,950. Technically, on weekly charts the index has formed a bar reversal candle which clearly suggests short term weakness will continue in the near future. However, the medium term texture of the market is still bullish and likely to continue if the nifty succeeds to trade above 14900-15000, which is earlier resistance level (Prior to the Budget of announcement). In addition, despite short term weakness the market is still trading well above short term and medium term averages which also support medium term uptrend. The market will continue to consolidate further as the Key index has entered below thr 15,000 level which was marked as the strong support by analysts. Traders might see some positive action if the key index will approach above 15,000 level, else the traders will witness more volatality.
As index managed to breached to its strong support of 15k mark which will acts as an immediate resistance now, so above 15k mark we may see some relief otherwise we may see more downside levels of 14,900-14,750 on the other hand 15,100-15,170 will act as a strong hurdle on the higher side. The index has made a bearish engulfing candlestick pattern which indicates price rejection at higher levels. The bulls are getting tired as the index is trading much higher than its mean levels and at an accelerated rising channel resistance. Hence, a brief corrective dip cannot be ruled out. Nifty50 has broken the immediate support of 15050 and a sustained price move below the support can trigger some more profit booking. Lack of any positive triggers may keep markets dull and range-bound. Fresh investments can be made on dips into quality bets as the market is in a longer term bull rally with currently intermediate top in the making.

From the high of 15431, the NIFTY has come off nearly 500-odd points. From the FO data, until expiry on 25FEB, we feel that some technical pullback is likely as the NIFTY has got deeply oversold in the immediate short term. It is unlikely that the NIFTY dips much below 15000 as the Index enjoys a good short term support at 14950-14900 levels. From the current levels, we expect some technical pullback to happen until 15200-odd levels. Fresh directional cues will appear only after that.

If Nifty sustains below 14900 in the coming week, the correction is likely to extend towards 14730-14600. However, if the index manages to hold above 14900 and goes on to surpass 15470, a fresh up leg can be expected to unfold that could take the index towards 15910.

25 Feb, 2021 Thursday

Sensex ends above 51K, Nifty near 15,100 (up by 115 points). Nifty ends the February F&O series near 15,100-mark. The Nifty formed a small Bodied candle on daily scale with long upper shadow which indicates that follow up is required to attempt the life time highs. It continues its formation of higher highs - higher lows of the last two trading sessions.Domestic market added strength to yesterday’s rally supported by positive F&O monthly rollovers, and robust global market. Small and mid-cap stocks continued its outperformance over the benchmark indices. It negated its formation of lower lows of the last four trading sessions but continues forming lower highs. Domestic market added strength to yesterday’s rally supported by positive F&O monthly rollovers, and robust global market. Small and mid-cap stocks continued its outperformance over the benchmark indices. Nifty has to continue to hold above 15,000 zones to extend its move towards 15,250 then 15,400 zones while on the downside immediate support exists at 14,900 and 14,850 levels

Nifty forms Shooting Star pattern on F&O expiry day. Except FMCG, all other sectoral indices ended in the green with metal index rising 4% and energy index adding 3%. Market continued to rally on February 25 on the back of positive global cues with Nifty ending the February F&O series near 15,100-mark. At close, Sensex was up 257.62 points, or 0.51%, at 51,039.31, and the Nifty was up 115.40 points, or 0.77%, at 15,097.40.

Despite a gap-up opening, the bulls appear to have failed to capitalise on the same as the Nifty witnessed intraday profit-booking from the highs of 15,176, which depicted a Shooting Star formation with a relatively long longer upper shadow. Hence, some intraday weakness can be expected if the Nifty slips below 15,065, though a short-term weakness can be expected on a close below 15,000, which can once again drag the index down into the sideways zone. A positive stance can be retained as long as the Nifty sustains above 15,000. If the bulls manage to push the index beyond 15,173, then the strength shall expand initially to 15,271, with eventual targets present around life highs present around 15,430. For the time, traders who are long are advised to hold their position with a stop below 15,000 on the closing basis.

Reliance Industries stood among the top index contributer as it surged over four per cent in the intraday trade to hit the day’s high of Rs 2,152.00. The India Volatility Index ended 5.3% lower at 22.89. 1,264 stocks on the NSE ended with gains while 622 posted losses.

Going ahead from here, from the current technical perspective, crossing of the previous day’s high, i.e., 15,176 will be crucial if Nifty has to extend its up move. Friday’s session is likely to see the levels of 15,175 and 15,210 acting as immediate resistance points, while support will come in at 15,050 and 14,970 levels.

The Relative Strength Index (RSI) on the daily chart stood neutral at 59.34; it did not show any divergence against price. The daily MACD was bearish and remained be .. A Shooting Star occurred on the charts. This is a classical Shooting Star as Nifty opened high, inched higher before closing more towards its low point of the day. This type of candle can probably form a temporary top again and push the market into consolidation. However, as always, this will require confirmation on the next trading day. Following two days of strong up move, chances are high that Nifty may slip into some consolidation. If the index extends its up move, it is likely to find strong resistance at 15,176 levels. From the options data as well, the level of 15,200 holds maximum Call OI as of now. From a technical perspective, the zone of 15,175-15,200 remains a critical resistance for the market to navigate in the immediate near-term.

MacOS BigSur Developer Setup

2021-02-01T00:00:00+00:00

Install MySQL server and workbench

MySql server is the de-facto standard for RDBMS requirementents

https://medium.com/macoclock/mysql-on-mac-getting-started-cecb65b78e http://g2pc1.bu.edu/~qzpeng/manual/MySQL%20Commands.htm https://vladster.net/en/instructions/install-mysql-mac/

// mysql server installer directory
/usr/local/mysql/bin/mysql -u root -p 

// set path for zsh
sudo sh -c 'echo /usr/local/mysql/bin > /etc/paths.d/mysql'
mysql --version
mysql -u root -p

Configuring Python environment

Anaconda offers the easiest developer friendly way to setup any system for data engineering. Anaconda is a package manager, an environment manager, and Python distribution that contains a collection of many open source packages. Anaconda Individual Edition contains conda and Anaconda Navigator, as well as Python and hundreds of scientific packages. When you installed Anaconda, you installed all these too. Notice that when you install Anaconda, it modifies your .bash_profile to put Anaconda in your path. https://opensource.com/article/19/5/python-3-default-mac

/user/bin/python
python --version
echo $PATH
which python
conda --version
jupyter notebook

Setting up & configuring hibernate modes

// Sleep mode and keep awake
pmset -g | grep hibernatemode
	hibernatemode 0  (normal sleep)
	hibernatemode 1  (hibernate mode for pre-2005 portable Macs)
	hibernatemode 3  (safe sleep)
	hibernatemode 25 (hibernate mode for post-2005 portable Macs

sudo pmset -a hibernatemode X

caffeinate -dt 7200

Version Check bash commands

mysql --version
conda --version
python --version
pip --version

Building real time data pipelines with AWS Kinesis

2020-09-12T00:00:00+00:00

Amazon Kinesis Data Streams is a fully managed and scalable AWS service which enables us to:

ingest, buffer and process data in real time

analyze real-time streaming data as it arrives 

respond instantly instead of having to wait until all your data is collected before the processing can begin

Introduction

Kinesis enable architectures to evolve from batch to real-time data processing. It can ingest real-time data such as logs and IoT telemetry data for machine learning and analytics. The producers continually push data to Kinesis Data Streams, and the consumers process the data in real time. The Consumers can store their results using an AWS service such as Amazon DynamoDB, Amazon Redshift, or S3. Producers assign partition keys to the records and ingest to the corresponding shards. Consumers retrieve the most recent data in a shard.

Kinesis data streams can be accessed privately with encryption enabled for better security. IT also provides enhanced fan-out feature that dedicatedly gives double throughput to the consumers. Producer and consumer applications can be built using efficient AWS libraries and services.

The unit of data stored by Kinesis Data Streams is a data record. A data stream represents a group of data records. The data records in a data stream are distributed into shards.

Official AWS Links:

Shards

A shard has a sequence of data records in a stream. When you create a stream, you specify the number of shards for the stream. The total capacity of a stream is the sum of the capacities of its shards. You can increase or decrease the number of shards in a stream as needed. A producer puts data records into shards and a consumer gets data records from shards. Shard is the base throughput unit of a Kinesis stream and supports maximum 1MB per second data rate.

• 200 records of 5kb each per second using 1 shard • 1GB or 0.2 million records per minute with 15 shards

AWS CLI Kinesis commands

KCL and KPL Java libraries for building producers and consumers

The Kinesis Producer Library (KPL) aggregates small user-formatted records into larger records up to 1 MB to make better use of Amazon Kinesis Data Streams throughput. While the KCL for Java supports deaggregating these records.

What is the Kinesis Client Library?

One problem, the KCL and KPL are heavily focused on Java, but most of the data scientists love Python. One can always create amazon-kinesis-client-python library on top of Java MultiLangDaemon for interprocess communication, but it is not recommended.

Writing a python kinesis producer

Writing a python kinesis consumer

Kinesis Data Streams Consumers

Python Boto3

get the list of existing kinesis streams
create a new kinesis stream
get the details of a new stream
publish to kinesis stream
monitor kinesis stream
consume records from the stream

Strategies with Heikin-Ashi patterns

2020-05-28T00:00:00+00:00

Heikin-Ashi means “average bar” in Japanese.

Introduction

Heiken-Ashi candles essentially captures the pace of price.

Market fluctuations, volatility and price oscilattion lead to the formation of alternating bullish and bearish candles. The movement makes it difficult to judge the trend. To overcome this, chartists use smoothing techniques and this is where Heiken Ashi shines. Heiken Ashi make trends easier to spot and help traders to remain longer in trade.

The Heikin-Ashi technique can be used in conjunction with candlestick charts when trading securities to spot market trends and predict future prices. It’s useful for making candlestick charts more readable and trends easier to analyze. For example, traders can use Heikin-Ashi charts to know when to stay in trades while a trend persists but get out when the trend pauses or reverses.

The resulting candlestick filters out some noise in an effort to better capture the trend. The time series is defined by the user, depending on the type of chart desired, such as daily, hourly or five-minute intervals.

The values used to construct Heiken Ashi candlesticks, are averages. Averaging helps to smooth out short-term price variations. Since it’s taking an average of the movement, the candlestick has a smoother look. These candlesticks can be used to identify trending periods, potential reversal points and classic technical analysis patterns.

How Heikin-Ashi is calculated

Heiken Ashi candlesticks are similar to conventional ones, but rather than using opens, closes, highs and lows, they use average values for these four price metrics.

Each of the Heikin Ashi bars start from the middle of the bar before it, and not from the level where the previous candle has closed. This is a major distinguishing factor between the standard and HA candlestick patterns.

Heikin-Ashi Candlesticks use the open-close data from the prior period and the open-high-low-close data from the current period to create a combo candlestick. The Heiken Ashi formula used to derive these average values is as follows:

Open = (open of previous bar + close of previous bar)/2
Close = (open + high + low + close)/4
High = the maximum value from the high, open, or close of the current period
Low = the minimum value from the low, open, or close of the current period

Heikin-Ashi Open price is a result of the previous Heikin-Ashi values. So the first HA candle is calculated using below standard solution:

HA_Close = (Open + High + Low + Close) / 4
HA_Open = (Open + Close) / 2
HA_Low = Low
HA_High = High

How to interprete and indentify trends using Heikin-Ashi patterns

Heikin-Ashi Candlesticks provide chartists with a versatile tool that can filter noise, foreshadow reversals and identify classic chart patterns. It support all aspects of classical technical analysis and charting. Chartists can use Heikin-Ashi Candlesticks to identify support and resistance, draw trend lines or measure retracements. Volume indicators and momentum oscillators also work well. Heikin Ashi charts are often used as a technical indicator on a typical candlestick chart, to help highlight and clarify the current trend.

Heikin Ashi charts are sometimes by swing traders or investors. Day traders tend to use Heikin Ashi charts more as an indicator, as HA charts have certain other benefits. There are five primary signals that identify trends and buying opportunities:

green candles signify an uptrend
green candles with no lower “shadows” indicate a strong uptrend
candles with a small body surrounded by upper and lower shadows indicate a trend change
red candles indicate a downtrend
red candles with no higher shadows identify a strong downtrend

A Heiken Ashi trader looks for two particular signals:

A green candle without a lower shadow, which is a particularly bullish signal
A red candle without an upper shadow, which is a particularly bearish signal

These signals may make locating trends or trading opportunities easier than with traditional candlesticks. The trends are not interrupted by false signals as often and are thus more easily spotted.

Common Heiken Ashi Candle Types & patterns

Size of body, shadows, and range of candle determines whether it is bullish, bearish or a neutral candle.

continuation candles - Small narrow range candles
strong up-trend candles - wide range candles with no tail
strong down-trend candles
indecision candle - small body, long tail and shadow on both sides. These represent Trend change or pause in Trend
Doji candle
Spinning top candle
Bearish candles - Such candles do not have any shadow

Some of the common patterns formed on the Heikin Ashi chart include: Doji Reversal Candlestick Triangle Patterns Rising Wedge and Falling Wedge Patterns

Initiation candle is one that sets the tone of Trend and defines underlying momentum for price. This is why Initiation candles are most important in Trend Analysis and Price action trading. Continuation candles are ones that reaffirm the direction of trend and are useful to increase positions in the direction of trend.

Heikin-Ashi doji and spinning tops

As with normal candlesticks, Heikin-Ashi doji and spinning tops can be used to foreshadow reversals. A Heikin-Ashi doji or Heikin-Ashi spinning top looks just the same as a normal doji or spinning top.

A doji is a small candlestick with an open and close that are virtually equal. There are small upper and lower shadows to denote little price movement.
Spinning tops have small bodies (open-close range) and long upper/lower shadows (high-low range). Despite a lot of movement from high to low, prices finish near their opening point for little change. This shows indecision that can foreshadow a reversal.

When using Heikin-Ashi candlesticks, a doji or spinning top in a downtrend should not immediately be considered bullish. It just shows indecision within the downtrend. Indecision is the first step to changing direction. Confirmation of a directional change (trend reversal) is required though. Once chartists spot a doji or spinning top in a downtrend, it is time to set a resistance level upon which to base a trend reversal.

Trade strategies with Heiken-Ashi

There are broadly 5 patterns that need to be followed when trading with Heiken Ashi Candles.

Green candles with no lower shadows indicate a strong uptrend: When you spot these on charts, be in the trade and don’t think about profit booking. You might want to add to your long position and exit short positions.
Candles with a small body with upper and lower shadows indicate trend change: These are indecision candles and require more confirmation.
Green/Red candles with no upper shadow indicates strong a downtrend: When you spot these on charts, be in the trade and don’t think about profit booking. You might want to add to your short position and exit long positions
Candles with long lower shadows represent Buying interest. Always take note of these candles and assess price action after you spot these candles.
Candles with long upper shadows represent selling interest and be cautious with existing long positions if you spot such Candles.

Every Heikin Ashi trading system should include price action rules:

Look for Support and Resistance levels.
Look for chart patterns for potential breakouts.
Hop into emerging trends and exit on slowing trends.

** Trailing Stop Method ** Using a trailing stop is a good technique to pursue in a trending market. Smoothing benefits of the Heikin Ashi chart can be combined with a trailing stop indicator to get the most out of a trending market condition.

** Heiken Ashi Trading Strategy For Short & Longer Time Frame Charts **

** Heiken Ashi Expansion Pattern **

** Heiken Ashi Double Bottom Momentum Pattern ** https://tradingwithtrend.com/heiken-ashi-strategy

Using Heiken Ashi along with momentum indicators, helps it easier to confirm the trend. For example, a Heiken Ashi moving average strategy might wait for the candles to cross over a 50-period moving average, as an entry signal.

You would enter once the next confirming candle occurs, that being a bullish candle, if you were waiting to buy after an upward crossover - and vice versa.

Similarly, you could use a Ichimoku cloud. Ichimoku cloud a.k.a Ichimoku Kinko Hyo, is a candlestick-based, trend-following system. It was originally designed with regular candlesticks in mind, but some traders use Heiken Ashi candles instead. This Ichimoku Heiken Ashi combination can enhance your strategy and make it easier to stick with the trend.

Ralph Lauren - Defining modern luxury and timeless style since 1967

2020-05-10T00:00:00+00:00

Founded by Ralph Lifshitz in 1967 with a current revenue of ‎US $163 million (2018)

Scala Snippets - Read/Write to a file

2018-09-23T00:00:00+00:00

Reading/Writing to a file - Scala I/O

Scala doesn’t provide any native class/library support when it comes to writing output to a file or reading a file. So we borrow java I/O class in scala, create a new instance of PrintWriter and pass a new File object to it:

import java.io._        // to write to the file

//create a new file
val writer = new PrintWriter(new File("demo1.txt"))

//call the method write() on the object we created 
writer.write("This is a demo")

//close it
writer.close()

concise way of writing a file in scala

import java.io.PrintWriter
new PrintWriter("filename") { write("file contents"); close }

while this is a one liner, it is unsafe. The approach looks nice, it is neither exception-safe nor encoding-safe. If an exception happens in write(), close will never be called, and the file won’t be closed.

PrintWriter also uses the default system encoding, which is very bad for portability. And finally, this approach generates a separate class specifically for this line. Scala already generates tons of classes even for a simple code.

Consuming a file in scala

//Read a file
//to read a file, call the fromFile() method of class Source, with the filename as argument
import scala.io.Source  // to read a file
Source.fromFile("demo1.txt").mkString   // reads into a single line
scala.io.Source.fromPath("file.txt").getLines.reduceLeft(_+_)

The above leaves the file open, however. To avoid problems, you should close it like this:

val source = scala.io.Source.fromFile("file.txt")
val lines = try source.mkString finally source.close()

Another problem with the code above is that it is horrible slow due to its implementation nature. For larger files one should use:

source.getLines mkString "\n"

Refrain from reading an entire file into a single String. The getLines method returns a value of type Iterator[String]. It’s effectively a lazy cursor into the file, allowing you to examine just the data you need without risking memory glut.

//to read individual lines instead of the whole file at once, use the getLines() method
Source.fromFile("demo1.txt").getLines.foreach{x=>println(x)}

//iterator can also be used to get one line at a time
val it = Source.fromFile("demo1.txt").getLines()
it.next()
it.next()

//method take(n) to return the first n values from the iterator
it.take(2)

//print all the lines using while loop
while(it.hasNext){ print(it.next()) }

//method slice(start,until) returns an iterator over lines start to until-1.
it.slice(1,5)

Transitioning to functional programming

2018-09-17T00:00:00+00:00

Common difference points in functional style of coding

Pure functions with no side effects - Given the same inputs, always returns the same output. This makes function calls completely independent of other function calls, which can radically simplify changes and refactoring. With impure functions, it’s impossible to fully understand what a function does unless you know the entire history of every variable that the function uses or affects. All useful Pure Functions must take at least one parameter and must return something. A pure function will always produce the same output given the same inputs with no side effects.

var z = 10;
function add(x, y) {
    return x + y;
}
console.log(add(1, 2)); // prints 3

All of the below function have what are called Side Effects and hence impure functions. When you call them, they change files and database tables, send data to a server or call the OS to get a socket. They do more than just operate on their inputs and return outputs. Therefore, you can never predict what these functions will return.

writeFile(fileName);
updateDatabaseTable(sqlCmd);
sendAjaxRequest(ajaxRequest);
openSocket(ipAddress);

Declarative rather than imperative - Functional programming is a declarative paradigm, meaning that the program logic is expressed without explicitly describing the flow control. Declarative programs abstract the flow control process, and instead spend lines of code describing the data flow: What to do. The how gets abstracted away.

Referential transparency - you can replace a function call with its resulting value without changing the meaning of the program

Function composition - process of combining two or more functions in order to produce a new function or perform some computation. For example, f.g or f(g(x))

Avoid shared state - relies on immutable data structures and pure calculations to derive new data from existing data hence eliminating the race condition. When you avoid shared state, the timing and order of function calls don’t change the result of calling the function.

A change in one function, or the timing of a function call won’t ripple out and break other parts of the program.

Immutability - An immutable object is an object that can’t be modified after it’s created. FP have special immutable data structures called trie data structures which are effectively deep frozen —, meaning that no property can change, regardless of the level of the property in the object hierarchy. There are no variables in Functional Programming. Stored values are still called variables because of history but they are constants, i.e. once x takes on a value, it’s that value for life.

// This is illegal in FP
var x = 1;
x = x + 1;

Side Effects - Any application state change that is observable outside the called function other than its return value

First class functions - allows us to treat functions as data — assign them to variables, pass them to other functions, return them from functions, etc

Higher order functions - any function which takes a function as an argument, returns a function, or both. The same map() utility can map over objects, strings, numbers, or any other data type because it takes a function as an argument which appropriately handles the given data type.

Functional programming favors:

Pure functions instead of shared state & side effects
Immutability over mutable data
Function composition over imperative flow control
Lots of generic, reusable utilities that use higher order functions to act on many data types instead of methods that only operate on their colocated data
Declarative rather than imperative code (what to do, rather than how to do it)
Expressions over statements
Containers & higher order functions over ad-hoc polymorphism

More Information pointers:

Functional Programming — A new paradigm

Master the JavaScript Interview: What is Functional Programming?

Master the JavaScript Interview: What is a Pure Function?

So You Want to be a Functional Programmer (Part 1)

All about Data Shuffling in Apache Spark

2018-08-28T00:00:00+00:00

Avoid shuffling at all cost. Think about ways to leverage existing partitions. Leverage partial aggregation to reduce data transfer.

Shuffling is a process of redistributing data across partitions (aka repartitioning) that may or may not cause moving data across JVM processes or even over the wire (between executors on separate machines). Shuffling is the process of data transfer between stages. By default, shuffling doesn’t change the number of partitions, but their content.

RDD is created by loading a file from HDFS, or reading a file in local storage. Spark has no control over what bits of data are distributed in which partitions.

The data is read and partitioned in an RDD, and when an “action” function is called, Spark sends out tasks to the worker nodes. If the action is a reduction, data shuffling takes place.

This becomes a problem for key-value RDDs: these often require knowing where occurrences of a particular key are, for instance to perform a join. If the key can occur anywhere in the RDD, we have to look through every partition to find the key.

Avoid groupByKey and use reduceByKey or combineByKey instead.

groupByKey shuffles all the data, which is slow.
reduceByKey shuffles only the results of sub-aggregations in each partition of the data.

PairRDD offers join transformation that when called on datasets of type (K, V) and (K, W), returns a dataset of (K, (V, W)) pairs with all pairs of elements for each key

when shuffling is triggered on Spark? Any join, cogroup, or ByKey operation involves holding objects in hashmaps or in-memory buffers to group or sort. join, cogroup, and groupByKey use these data structures in the tasks for the stages that are on the fetching side of the shuffles they trigger.

distinct creates a shuffle

reduceByKey and aggregateByKey use data structures in the tasks for the stages on both sides of the shuffles they trigger.

How to check for shuffle?

toDebugString returns “A description of this RDD and its recursive dependencies for debugging.” So it will include possible shuffles from prior transformations if they exist, even if the most recent transformation does not incur shuffle.

scala> val a  = sc.parallelize(Array(1,2,3)).distinct
scala> a.toDebugString
MappedRDD[5] at distinct at <console>:12 (1 partitions)
  MapPartitionsRDD[4] at distinct at <console>:12 (1 partitions)
    **ShuffledRDD[3] at distinct at <console>:12 (1 partitions)**
      MapPartitionsRDD[2] at distinct at <console>:12 (1 partitions)
        MappedRDD[1] at distinct at <console>:12 (1 partitions)
          ParallelCollectionRDD[0] at parallelize at <console>:12 (1 partitions)

There are situations where a shuffle will be required or not required for a certain function. For example join usually requires a shuffle but if you join two RDD’s that branch from the same RDD, spark can sometimes elide the shuffle.

Serialization in Spark

2018-08-25T00:00:00+00:00

Serialization is implemented in most distributed applications for performance. A serialization framework helps you convert objects into a stream of bytes and vice versa in new computing environment. This is very helpful when you try to save objects to disk or send them through networks. Those situations happen in Spark when things are shuffled around. RDDs can be stored in serialized form, to decrease memory usage, reduce network bottleneck and performance tuning.

Java serialization
Kryo serialization

Under the hood, a dataset is an RDD. Store RDD as serialized Java objects (one byte array per partition). This is generally more space-efficient than deserialized objects, especially when using a fast serializer, but more CPU-intensive to read. By default, Java serialization is used. To enable Kryo, initialize the job with a SparkConf and set spark.serializer to org.apache.spark.serializer.KryoSerializer

val conf = new SparkConf()
.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.set("spark.kryoserializer.buffer.mb","24")

val sc = new SparkContext(conf)

Every task run from Driver to Worker gets serialized : Closure serialization
Every result from every task gets serialized at some point : Result serialization

And what’s implied is that during all closure serializations all the values used inside will get serialized as well, for the record, this is also one of the main reasons to use Broadcast variables when closures might get serialized with big values.

There are three considerations in tuning memory usage:

the amount of memory used by your objects (you may want your entire dataset to fit in memory)
the cost of accessing those objects,
the overhead of garbage collection (if you have high turnover in terms of objects).

In apache spark, it’s advised to use the kryo serialization over java serialization for big data applications. Kryo has less memory footprint compared to java serialization which becomes very important when you are shuffling and caching large amount of data.

Eradication the most common serialization issue:

This happens whenever Spark tries to transmit the scheduled tasks to remote machines. Tasks are just pieces of application code that are sent from the driver to the workers. The classpath of the driver and worker nodes are controlled by the user that is launching the application.

org.apache.spark.SparkException: Task not serializable
        at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:304)
        at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:294)
        at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:122)
        at org.apache.spark.SparkContext.clean(SparkContext.scala:2055)
        at org.apache.spark.rdd.RDD$$anonfun$map$1.apply(RDD.scala:324)
        at org.apache.spark.rdd.RDD$$anonfun$map$1.apply(RDD.scala:323)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
        at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
        at org.apache.spark.rdd.RDD.map(RDD.scala:323)

Ideally something is not Serializable and that threw the issue. Some basic Guidelines made by Databricks to avoid the scenario -

Make the object/class serializable
declare the instance within the lambda function.
Declare functions inside an Object as much as possible
If you need to use SparkContext or SQLContext inside closures (e.g. inside foreachRDD), then use SparkContext.get() and SQLContext.getActiveOrCreate() instead
Redefine variables provided to class constructors inside functions

Life is a File 📁

Daily Market Analysis - March 2021

01 Mar, 2021 Monday

Nifty50

Daily Market Analysis - February 2021

01 Feb, 2021 Monday | Budget Day

02 Feb, 2021 Tuesday

03 Feb, 2021 Wednesday

04 Feb, 2021 Thursday

05 Feb, 2021 Friday

08 Feb, 2021 Monday

09 Feb, 2021 Tuesday

10 Feb, 2021 Wednesday

11 Feb, 2021 Thursday

12 Feb, 2021 Friday

Nifty50

Nifty Bank

15 Feb, 2021 Monday

Nifty50

16 Feb, 2021 Tuesday

Nifty50

18 Feb, 2021 Thursday (weekly expiry)

Nifty50

19 Feb, 2021 Friday

Stock Market performance this week

Reason behind the market correction

Where could market possibly head from here next week

25 Feb, 2021 Thursday

MacOS BigSur Developer Setup

Install MySQL server and workbench

Configuring Python environment

Setting up & configuring hibernate modes

Version Check bash commands

Building real time data pipelines with AWS Kinesis

Introduction

Official AWS Links:

Shards

AWS CLI Kinesis commands

KCL and KPL Java libraries for building producers and consumers

Writing a python kinesis producer

Writing a python kinesis consumer

Python Boto3

Strategies with Heikin-Ashi patterns

Introduction

How Heikin-Ashi is calculated

How to interprete and indentify trends using Heikin-Ashi patterns

Common Heiken Ashi Candle Types & patterns

Heikin-Ashi doji and spinning tops

Trade strategies with Heiken-Ashi

Ralph Lauren - Defining modern luxury and timeless style since 1967

Scala Snippets - Read/Write to a file

Reading/Writing to a file - Scala I/O

Transitioning to functional programming

Common difference points in functional style of coding

Functional programming favors:

More Information pointers:

All about Data Shuffling in Apache Spark

How to check for shuffle?

Serialization in Spark

Eradication the most common serialization issue:

Pointers to follow: