The Impact of Shredded Storage on SharePoint 2013

The Impact of Shredded Storage on SharePoint 2013

By Trevor Hellebuyck | February 19, 2013

Read Part 2Dispelling the Myths of Shredded Storage in SharePoint 2013

There is little doubt the SharePoint community is excited for SharePoint 2013. With 60% of users in a recent SharePoint survey saying they want to upgrade in the next year, the anticipation is building to a climax.

One feature that has garnered a lot of buzz, and some confusion, is the new Shredded Storage feature and what impact it will have on binary large objects (BLOBs) storage. In this two-part series, we’ll delve deeper into Shredded Storage, explore its impact on SharePoint 2013 and address the myths about the new feature and Remote BLOB Storage.

Before we plunge into the details it makes sense to define what Shredded Storage is and is not. The term "shredding" has created a lot of confusion. Shredded Storage does not refer to file shredding, which is the secure deletion of files by overwriting the data multiple times. StoragePoint and many other storage-related products include this feature. Instead, SharePoint 2013’s Shredded Storage is an attempt by Microsoft to reduce the I/O impact when saving versions of a document or file by "shredding" it into smaller pieces and reassembling it when someone needs to access it.

Microsoft developed shredded storage to address an issue that resulted from how document edits or versions are stored in SharePoint and to reduce the number of transaction logs. The result – significantly improved I/O (network and disk) and reduced CPU overhead when storing incremental changes to documents. The SharePoint product team has done a fantastic job addressing what has been a long-standing complaint in the way files are stored in previous versions of SharePoint.

Unfortunately, by addressing one issue, Microsoft introduced another that results in a performance decrease for the uploading and downloading of files in SharePoint.

A Brief History Lesson

With the introduction of SharePoint Team Services and subsequently SharePoint Portal Server 2003, Microsoft moved away from using the Web Storage System and settled on using SQL Server exclusively for the storage of BLOBs. BLOBs are immutable objects when stored in SharePoint. This means that BLOBs are created and deleted but never updated. When editing existing documents in SharePoint, edits result in new BLOBs being created. These new BLOBs are full copies including the edits and not incremental changes.

This means that if you maintain 10 versions of a 1MB document you would end up with approximately 10MB in total storage requirement excluding metadata. While this wasn't the most efficient use of storage it was the simplest way to handle versions without introducing complexity. Unfortunately SharePoint has a tendency to spin off new versions of documents even if only metadata has changed. With the introduction of SharePoint and Office 2010 Microsoft optimized communication between the Office client and the SharePoint server by implementing a file synchronization protocol named Cobalt.

What Does Cobalt Mean?

I am not going to cover Cobalt in nauseating detail as the topic has been covered quite well by Bill Baer in his post about Shredded Storage. What you should know is that Cobalt allowed the Office client to send incremental changes rather than the entire file to the server when a document was saved. The incremental changes were then reassembled on the server and saved as a complete file. Shredded Storage in SharePoint 2013 extends the saving of incremental changes to the database where only file changes are stored rather than entire copies of the file.

The goal of Cobalt and subsequently Shredded Storage is to reduce the network bandwidth utilization (client changes sent to the SharePoint server) and the I/O operations (SharePoint Server sends file to database) that result from incremental changes to documents. In fact Shredded Storage significantly improves I/O operations while reducing CPU overhead when saving incremental changes to documents. What isn't immediately apparent is the negative impact to I/O operations for the inserting of new files and downloading of existing files.

The Test Results

The table below compares SharePoint 2010 and SharePoint 2013 upload and download times on the same document set. Our lab testing confirms that SharePoint 2013 uploads and downloads are slower — and in some cases significantly slower — than SharePoint 2010. This is a direct result of Shredded Storage. The overhead involved in determining how to split a document into smaller pieces and store those smaller pieces definitely has an impact on the performance for uploads and downloads.

 

 

 

 

 

Upload (speed in milliseconds)

 

 

Download (speed in milliseconds)

 

 

Scenarios

File Name

File Type

File Size (KB)

SP2010   (A)

SP2013 (B)

Difference (A-B)

SP2010        ( C)

SP 2013 (D)

Delta (C-D)

1

AA_Small TIF

TIF

60

0.58

0.25

0.33

0.02

0.03

-0.01

2

AB_PDF Sample

PDF

625

0.11

0.39

-0.29

0.02

0.05

-0.03

3

AC_SharePoint Training

PPTX

669

0.15

0.72

-0.57

0.02

0.12

-0.10

4

AD_Drawing1

VSD

759

0.16

0.47

-0.31

0.02

0.05

-0.03

5

AE_1 MB Word Doc 2010

DOCX

1,082

0.35

0.66

-0.30

0.03

0.10

-0.07

6

AF_LV111-01-10

DWG

1,208

0.15

0.55

-0.41

0.03

0.07

-0.03

7

AG_1 mb image

JPG

1,210

0.20

0.64

-0.43

0.03

0.07

-0.04

8

AH_Drawing2

VSD

1,659

0.24

0.78

-0.54

0.04

0.09

-0.05

9

AI_Customer 2009

PPT

2,192

0.34

0.93

-0.59

0.05

0.10

-0.06

10

AJ_2mb TIF Image

TIF

2,579

0.32

1.01

-0.69

0.05

0.12

-0.07

11

AK_2mb Image

JPG

2,725

0.34

1.06

-0.72

0.06

0.14

-0.08

12

AL_LV111-02-10

DXF

2,783

0.33

1.10

-0.77

0.06

0.16

-0.10

13

AM_3_6mb PDF Sample

PDF

3,690

0.49

1.47

-0.98

0.07

0.21

-0.13

14

AN_4 MB PDF

PDF

4,078

0.50

1.60

-1.10

0.08

0.19

-0.11

15

AO_Corporate Presentation 2007

PPT

4,248

0.49

1.69

-1.21

0.08

0.20

-0.11

16

AP_Analyst Briefing - 2008

PPT

4,434

0.54

1.68

-1.15

0.08

0.18

-0.10

17

AQ_4_5 mb Video

MOV

4,627

0.46

1.77

-1.31

0.10

0.19

-0.09

18

AR_4_5 mb wmv video

WMV

4,680

0.51

1.81

-1.30

0.24

0.18

0.06

19

AS_Internet Safety Presentation

PPT

4,839

0.42

1.84

-1.42

0.24

0.20

0.05

20

AT_5mb Image

JPG

5,267

0.50

2.15

-1.66

0.21

0.23

-0.02

21

AU_LV111-01-10

DXF

5,425

0.42

2.02

-1.60

0.28

0.25

0.03

22

AV_5_3 JPG

JPG

5,457

0.55

2.08

-1.53

0.19

0.23

-0.04

23

AW_LV111-02-FL

DXF

5,866

0.48

2.11

-1.63

0.24

0.22

0.02

24

AX_Corporate Slide Deck_April 2009

PPT

5,936

0.53

2.29

-1.76

0.18

0.27

-0.09

25

AY_LV111-01-FL

DXF

5,972

0.44

2.22

-1.78

0.13

0.27

-0.14

26

AZ_7mb Excel File

XLSX

7,415

0.68

0.89

-0.22

0.33

0.28

0.05

27

BA_SPC14_348_WhatsNewDevs

PPTX

8,935

0.76

1.55

-0.80

0.30

0.95

-0.65

28

BB_SPC 2009

PPT

9,255

0.89

3.33

-2.45

0.39

0.34

0.04

29

BC_11_7mb Excel File

XLSX

11,974

0.92

1.25

-0.33

0.73

0.32

0.41

30

BD_14_5 MB PDF

PDF

14,861

1.24

5.08

-3.85

0.73

1.10

-0.38

31

BE_26 MB XLSX

XLSX

26,557

2.24

2.79

-0.54

1.31

2.29

-0.97

32

BF_28MB_txt_TestFile

TXT

28,787

2.05

8.58

-6.53

0.82

2.32

-1.49

33

BG_33_1 MB WORD 2010 Doc

DOCX

33,947

2.53

3.24

-0.71

0.62

3.02

-2.40

34

BH_50MB_txt_TestFile

TXT

54,265

3.69

15.92

-12.23

0.83

3.73

-2.90

35

BI_55 MB XLSX

XLSX

56,356

4.18

4.90

-0.72

2.29

5.48

-3.19

36

BJ_70 MB WORD 2010 Doc

DOCX

71,694

5.39

5.99

-0.59

6.55

6.36

0.19

37

BK_100 MB XLSX

XLSX

103,108

8.85

8.16

0.69

5.83

7.86

-2.03

38

BL_103 MB WORD 2010 Doc

DOCX

105,411

7.78

7.91

-0.13

4.84

7.20

-2.36

39

BM_180MB_txt_TestFile

TXT

184,288

13.52

10.53

2.98

13.55

5.82

7.72

40

BN_190 mb Word Doc 2003

DOC

195,899

14.65

12.01

2.63

14.30

8.22

6.08

41

BO_250 mb Movie

MOV

255,454

20.30

15.70

4.60

20.35

10.22

10.14

42

BP_382 mb Word Doc 2003

DOC

391,739

29.25

26.70

2.55

12.22

16.76

-4.55

43

BQ_540MB_txt_TestFile

TXT

552,862

45.21

42.21

3.00

31.71

17.36

14.35

 

One Size Does Not Fit All

The test results above led us to examine the configuration options for Shredded Storage to determine if we could mitigate the negative impact on uploads and downloads. Unfortunately your options are limited. Contrary to other blog posts on the topic, Shredded Storage cannot be disabled. You actually had the option to disable shredding in the SharePoint 2013 beta but that option was eliminated in the RTM build. The only remaining option is changing the default shred or "chunk" size that files will split into when they are stored.

For me the decision to disable shredding is a bit nearsighted. Not all organizations use SharePoint for document collaboration where content is being updated/edited in large quantities. I would even argue that while some organizations do have collaboration sites where lots of editing occurs, they almost certainly have other sites where documents are simply uploaded and downloaded without edits or new versions being created.

A common example is document imaging where PDF/TIFF images are stored within SharePoint. Those images never change. Or, how about a document center that contains tens of thousands of published documents that are being read rather than updated? What's more, Shredded Storage provides little value for these scenarios. It is true that even with versioning disabled the I/O between the client, SharePoint Server, and database Server will be optimized. However you will not reduce overall storage requirements.

Unfortunately you are relegated to living with Shredded Storage in hopes that Microsoft will provide, at a minimum the ability to disable the feature. An even better would be an option to control Shredded Storage at the site or site collection level for added flexibility.

Solving one problem by introducing another significant problem is going to make for some unhappy campers who are already struggling to keep up with the explosive growth of their SharePoint content.

In part 2, we will address using RBS with Shredded Storage, including debunking myths, reviewing how RBS functions with Shredding Storage, and discussing best practices for optimizing RBS.


Trevor Hellebuyck

Trevor is a recognized SharePoint innovator and the principal architect of StoragePoint, the ground-breaking storage optimization software for SharePoint. Trevor joined Metalogix in 2010 with the acquisition of BlueThread Technologies. He was instrumental in the launch and growth of StoragePoint while serving as Chief Operating Officer (COO) of BlueThread Technologies, a company focused on developing applications for Microsoft products and technologies. StoragePoint has become the #1 Remote BLOB Storage (RBS) solution in the market. Prior to BlueThread, Trevor led technology teams in Enterprise Content Management (ECM) and Enterprise Application Integration at NuSoft Solutions, acquired by RCM Technologies in 2008. 

Written By: Trevor Hellebuyck

Leave a Comment

Add new comment